fw-dataset

fw-dataset provides the classes and functions used to create, manage, and serve Flywheel Datasets — a way to organize, share, and query data derived from the Flywheel Data Model.

[!WARNING] This package is under active development and should be considered unstable. It is provided as-is, with no guarantee of support or maintenance at this stage. Features may be incomplete, change without notice, or be removed in future versions. Use it at your own risk, for experimental or development purposes only.

Overview

A Flywheel Dataset is a versioned, columnar snapshot of a Flywheel project's data, stored in cloud or local object storage and queryable with standard tabular tools. The package exposes three primary entry points:

• DatasetBuilder — renders a dataset from a Flywheel project snapshot.
• FWDatasetClient — accesses, queries, and manages existing datasets.
• Dataset / Table — object models representing a dataset and its tables.

Documentation

The Dataset Definition document is the authoritative reference for the dataset format. It covers:

• Dataset components and on-disk structure
• How datasets are rendered, including primary and derived tables
• Dataset storage layout
• A worked example structure and the current primary table schemas

Installation

fw-dataset requires Python 3.12 or later. Install it with pip:

pip install fw-dataset

Usage

Rendering datasets

Use DatasetBuilder to render a Flywheel dataset from a project. See notebooks/quickstart_dataset_creation.ipynb for a complete walkthrough.

Accessing and managing datasets

Use FWDatasetClient to access and query an existing dataset. See notebooks/quickstart_dataset_management.ipynb for a complete walkthrough.

Working with unassociated datasets

A valid dataset that is not associated with a Flywheel project can still be accessed directly. Provide the type, bucket, prefix, and credentials of the cloud or local filesystem to instantiate and query it — no API key or client instantiation required:

from fw_dataset import FWDatasetClient

fs_type = "s3"  # or "gcs", "azure", "fs", "local"
bucket = "your-bucket"
prefix = "your-prefix"
credentials = {"url": "{bucket-specific-credential-string}"}

dataset = FWDatasetClient.get_dataset_from_filesystem(fs_type, bucket, prefix, credentials)

Merging related datasets

Multiple datasets with related tables can be merged into a single dataset so their tables can be queried together.

[!NOTE] Federated querying across datasets is not yet enabled; this is a work in progress.

Merging requires that both datasets satisfy the following:

1. A valid tables directory structure.

2. A valid schemas directory structure, where:

   • every table in tables has a corresponding schema file in schemas;

   • each schema file is named {table_name}.schema.json; and

   • each schema file is valid JSON with at least the following structure:

     {
         "schema": "http://json-schema.org/draft-07/schema#",
         "id": "{table_name}",
         "description": "",
         "properties": {},
         "required": [],
         "type": "object"
     }
     

3. Tables and schemas selected from the source must not share names with existing tables or schemas in the destination.

Once these requirements are met, merge the datasets by copying or moving the selected tables and schemas from the source dataset into the destination dataset.

Flywheel project requirements

For the Flywheel Dataset client and the Dataset objects to function, a project must provide valid custom-information metadata and a well-formed storage layout.

Project metadata

The Flywheel project must carry the following custom information:

{
    "dataset": {
        "type": "s3",
        "bucket": "{bucket-name}",
        "prefix": "{path/to/dataset}",
        "storage_id": "storage-id-of-fw-storage-object"
    }
}

The directory structure beneath prefix must follow the Dataset structure described below.

Dataset structure

A dataset is stored beneath its prefix with the following layout:

{bucket}/{prefix}/
└── versions/
    └── {version}/
        ├── provenance/
        │   ├── dataset_description.json
        │   ├── snapshot.db.gz
        │   ├── snapshot_info.json
        │   └── project.json
        ├── tables/
        │   └── {table_name}/
        │       └── {hash}.parquet
        └── schemas/
            └── {table_name}.schema.json

Each version lives in its own subdirectory, named with its version identifier (typically a BSON ID such as 66cf6701af1c6f3855f1ee61). The "latest" version is determined dynamically by comparing the creation dates recorded in each version's dataset_description.json.

This layout is described more completely in the Dataset definition document.

Schema files

Schema files describe the schema of each table and live in the schemas directory. Each is named {table_name}.schema.json. Schemas should ideally be fully descriptive, but a minimal schema is sufficient to make a table queryable:

{
    "schema": "http://json-schema.org/draft-07/schema#",
    "id": "{table_name}",
    "description": "Table derived from tabular data file: conditions.csv",
    "properties": {},
    "required": [],
    "type": "object"
}