fw-dataset
fw-dataset provides the classes and functions used to create, manage, and serve
Flywheel Datasets — a way to organize, share, and query data derived from the
Flywheel Data Model.
Overview
A Flywheel Dataset is a versioned, columnar snapshot of a Flywheel project's data, stored in cloud or local object storage and queryable with standard tabular tools. The package exposes three primary entry points:
DatasetBuilder— renders a dataset from a Flywheel project snapshot.FWDatasetClient— accesses, queries, and manages existing datasets.Dataset/Table— object models representing a dataset and its tables.
Documentation
The Dataset Definition document is the authoritative reference for the dataset format. It covers:
- Dataset components and on-disk structure
- How datasets are rendered, including primary and derived tables
- Dataset storage layout
- A worked example structure and the current primary table schemas
Installation
fw-dataset requires Python 3.12 or later. Install it with pip:
pip install fw-dataset
Usage
Rendering datasets
Use DatasetBuilder to render a Flywheel dataset from a project. See
notebooks/quickstart_dataset_creation.ipynb
for a complete walkthrough.
Accessing and managing datasets
Use FWDatasetClient to access and query an existing dataset. See
notebooks/quickstart_dataset_management.ipynb
for a complete walkthrough.
Working with unassociated datasets
A valid dataset that is not associated with a Flywheel project can still be accessed
directly. Provide the type, bucket, prefix, and credentials of the cloud or
local filesystem to instantiate and query it — no API key or client instantiation
required:
from fw_dataset import FWDatasetClient
fs_type = "s3" # or "gcs", "azure", "fs", "local"
bucket = "your-bucket"
prefix = "your-prefix"
credentials = {"url": "{bucket-specific-credential-string}"}
dataset = FWDatasetClient.get_dataset_from_filesystem(fs_type, bucket, prefix, credentials)
Merging related datasets
Multiple datasets with related tables can be merged into a single dataset so their tables can be queried together.
Merging requires that both datasets satisfy the following:
-
A valid
tablesdirectory structure. -
A valid
schemasdirectory structure, where:-
every table in
tableshas a corresponding schema file inschemas; -
each schema file is named
{table_name}.schema.json; and -
each schema file is valid JSON with at least the following structure:
{ "schema": "http://json-schema.org/draft-07/schema#", "id": "{table_name}", "description": "", "properties": {}, "required": [], "type": "object" }
-
-
Tables and schemas selected from the
sourcemust not share names with existing tables or schemas in thedestination.
Once these requirements are met, merge the datasets by copying or moving the selected
tables and schemas from the source dataset into the destination dataset.
Flywheel project requirements
For the Flywheel Dataset client and the Dataset objects to function, a project must
provide valid custom-information metadata and a well-formed storage layout.
Project metadata
The Flywheel project must carry the following custom information:
{
"dataset": {
"type": "s3",
"bucket": "{bucket-name}",
"prefix": "{path/to/dataset}",
"storage_id": "storage-id-of-fw-storage-object"
}
}
| Field | Description |
|---|---|
type |
Storage backend: s3, gcs, azure, or fs/local. |
bucket |
Name of the bucket or container holding the dataset. |
prefix |
Path to the dataset within the bucket or container. |
storage_id |
Flywheel ID of the storage record for the bucket or filesystem. |
The directory structure beneath prefix must follow the Dataset
structure described below.
Dataset structure
A dataset is stored beneath its prefix with the following layout:
{bucket}/{prefix}/
└── versions/
└── {version}/
├── provenance/
│ ├── dataset_description.json
│ ├── snapshot.db.gz
│ ├── snapshot_info.json
│ └── project.json
├── tables/
│ └── {table_name}/
│ └── {hash}.parquet
└── schemas/
└── {table_name}.schema.json
Each version lives in its own subdirectory, named with its version identifier
(typically a BSON ID such as 66cf6701af1c6f3855f1ee61). The "latest" version is
determined dynamically by comparing the creation dates recorded in each version's
dataset_description.json.
This layout is described more completely in the Dataset definition document.
Schema files
Schema files describe the schema of each table and live in the schemas directory.
Each is named {table_name}.schema.json. Schemas should ideally be fully descriptive,
but a minimal schema is sufficient to make a table queryable:
{
"schema": "http://json-schema.org/draft-07/schema#",
"id": "{table_name}",
"description": "Table derived from tabular data file: conditions.csv",
"properties": {},
"required": [],
"type": "object"
}
Metadata
Release files for fw-dataset 0.4.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| fw_dataset-0.4.4-py3-none-any.whl | Python 3 | none | any | Details for fw_dataset-0.4.4-py3-none-any.whl |
Release files / fw_dataset-0.4.4-py3-none-any.whl
| Download URL | fw_dataset-0.4.4-py3-none-any.whl |
|---|---|
| Size | 59.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5daf86c27ac6f9c4cf510e6d21aabf4eaf50e9372b472b954317b975b089a048
|
|
BLAKE2b-256 checksum How to use checksums |
ff3dec5926ec1a9838cdadc1e235d9777189a2347bbd531532320caf51d2f922
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Alpine Linux","version":"3.25.0_alpha20260805","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for fw-dataset 0.4.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.