Skip to main content

fw-dataset

fw-dataset provides the classes and functions used to create, manage, and serve Flywheel Datasets — a way to organize, share, and query data derived from the Flywheel Data Model.

Overview

A Flywheel Dataset is a versioned, columnar snapshot of a Flywheel project's data, stored in cloud or local object storage and queryable with standard tabular tools. The package exposes three primary entry points:

  • DatasetBuilder — renders a dataset from a Flywheel project snapshot.
  • FWDatasetClient — accesses, queries, and manages existing datasets.
  • Dataset / Table — object models representing a dataset and its tables.

Documentation

The Dataset Definition document is the authoritative reference for the dataset format. It covers:

Installation

fw-dataset requires Python 3.12 or later. Install it with pip:

pip install fw-dataset

Usage

Rendering datasets

Use DatasetBuilder to render a Flywheel dataset from a project. See notebooks/quickstart_dataset_creation.ipynb for a complete walkthrough.

Accessing and managing datasets

Use FWDatasetClient to access and query an existing dataset. See notebooks/quickstart_dataset_management.ipynb for a complete walkthrough.

Working with unassociated datasets

A valid dataset that is not associated with a Flywheel project can still be accessed directly. Provide the type, bucket, prefix, and credentials of the cloud or local filesystem to instantiate and query it — no API key or client instantiation required:

from fw_dataset import FWDatasetClient

fs_type = "s3"  # or "gcs", "azure", "fs", "local"
bucket = "your-bucket"
prefix = "your-prefix"
credentials = {"url": "{bucket-specific-credential-string}"}

dataset = FWDatasetClient.get_dataset_from_filesystem(fs_type, bucket, prefix, credentials)

Multiple datasets with related tables can be merged into a single dataset so their tables can be queried together.

Merging requires that both datasets satisfy the following:

  1. A valid tables directory structure.

  2. A valid schemas directory structure, where:

    • every table in tables has a corresponding schema file in schemas;

    • each schema file is named {table_name}.schema.json; and

    • each schema file is valid JSON with at least the following structure:

      {
          "schema": "http://json-schema.org/draft-07/schema#",
          "id": "{table_name}",
          "description": "",
          "properties": {},
          "required": [],
          "type": "object"
      }
      
  3. Tables and schemas selected from the source must not share names with existing tables or schemas in the destination.

Once these requirements are met, merge the datasets by copying or moving the selected tables and schemas from the source dataset into the destination dataset.

Flywheel project requirements

For the Flywheel Dataset client and the Dataset objects to function, a project must provide valid custom-information metadata and a well-formed storage layout.

Project metadata

The Flywheel project must carry the following custom information:

{
    "dataset": {
        "type": "s3",
        "bucket": "{bucket-name}",
        "prefix": "{path/to/dataset}",
        "storage_id": "storage-id-of-fw-storage-object"
    }
}
Field Description
type Storage backend: s3, gcs, azure, or fs/local.
bucket Name of the bucket or container holding the dataset.
prefix Path to the dataset within the bucket or container.
storage_id Flywheel ID of the storage record for the bucket or filesystem.

The directory structure beneath prefix must follow the Dataset structure described below.

Dataset structure

A dataset is stored beneath its prefix with the following layout:

{bucket}/{prefix}/
└── versions/
    └── {version}/
        ├── provenance/
        │   ├── dataset_description.json
        │   ├── snapshot.db.gz
        │   ├── snapshot_info.json
        │   └── project.json
        ├── tables/
        │   └── {table_name}/
        │       └── {hash}.parquet
        └── schemas/
            └── {table_name}.schema.json

Each version lives in its own subdirectory, named with its version identifier (typically a BSON ID such as 66cf6701af1c6f3855f1ee61). The "latest" version is determined dynamically by comparing the creation dates recorded in each version's dataset_description.json.

This layout is described more completely in the Dataset definition document.

Schema files

Schema files describe the schema of each table and live in the schemas directory. Each is named {table_name}.schema.json. Schemas should ideally be fully descriptive, but a minimal schema is sufficient to make a table queryable:

{
    "schema": "http://json-schema.org/draft-07/schema#",
    "id": "{table_name}",
    "description": "Table derived from tabular data file: conditions.csv",
    "properties": {},
    "required": [],
    "type": "object"
}

Metadata

Release files for fw-dataset 0.4.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for fw-dataset 0.4.4
File Interpreter ABI Platform Actions
fw_dataset-0.4.4-py3-none-any.whl Python 3 none any Details for fw_dataset-0.4.4-py3-none-any.whl

Release files / fw_dataset-0.4.4-py3-none-any.whl

Download URL fw_dataset-0.4.4-py3-none-any.whl
Size 59.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5daf86c27ac6f9c4cf510e6d21aabf4eaf50e9372b472b954317b975b089a048
BLAKE2b-256 checksum
How to use checksums
ff3dec5926ec1a9838cdadc1e235d9777189a2347bbd531532320caf51d2f922
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Alpine Linux","version":"3.25.0_alpha20260805","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for fw-dataset 0.4.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.4 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page