Simple data catalog library for python

This project is a trivial attempt at offering basic catalog functionality for structured datasets stored in local or remote folders. The library uses universal_pathlib to access remote storage locations like S3, Google Cloud Storage, etc ... The library reads a config file called fsdata.ini which defines a list of collections, one per section. Each collection corresponds to a local or remote folder containing data files, homogeneous in format: parquet collections hold DataFrames (pandas or polars), json collections hold plain lists/dicts — declared per collection in the config (format = json; parquet is the default). Consumers pick the accessor for the shape they want (.pandas(), .polars(), .json()); the wrong accessor for the collection format raises. The library uses local caching to avoid fetching the same data multiple times.

Warning This project is for exploration only, the interface can change.

Configuration

The configuration file fsdata.ini has one section for each collection, with the section name for name and with a path key pointing to its location. The config file should be located in the the standard XDG config directory XDG_CONFIG_HOME (or ~/.config).

Each collection declares its format with a format key (parquet or json); parquet is the default when the key is omitted.

# fsdata.ini

[samples]
path = s3://my-bucket/samples

[datasets]
path = s3://my-bucket/datasets
format = parquet

[tickers]
path = s3://my-bucket/tickers
format = json

Usage

To access a collection use the collection function, or — for collection names that are valid identifiers — plain attribute access on the module.

import fsdata

samples = fsdata.collection("samples")
samples = fsdata.samples                  # same thing

To list the configured collections

fsdata.collection_names()

To list items in a collection (item names are bare names, without extension)

samples.items()
samples.has("my-sample")

To load data, pick the accessor for the shape you want. Each accessor has a single concrete return type, and raises if the collection format does not match.

samples.pandas("my-sample")     # -> pandas.DataFrame   (parquet collections)
samples.polars("my-sample")     # -> polars.DataFrame   (parquet collections)

tickers = fsdata.tickers
tickers.json("DOW30")           # -> plain list or dict (json collections)

To save data use the save method — the object type must match the collection format: pandas/polars DataFrames go to parquet collections, plain lists and dicts go to json collections. Anything else raises.

samples.save("my-sample", df)          # DataFrame -> .parquet
tickers.save("DOW30", ["MMM", "AXP"])  # list -> .json

Deprecated APIs

The following functions still work but emit a DeprecationWarning; new code should use the replacements.

The backend= parameter is superseded by the accessor names: instead of selecting the return type with an argument, call the accessor that returns what you want.

Installation

You can install the package with pip

pip install fsdata

You can specify any of the extra dependencies s3, gcs, adl to install the required fsspec backends.

pip install "fsdata[s3]"

Requirements

• pandas and/or polars (each needed only by its own accessor)
• pyarrow
• universal_pathlib
• fsspec backends like s3fs, etc ... as applicable

Related Projects and Resources

• intake - Lightweight package for finding, investigating, loading and disseminating data.
• pins - Publish data sets, models, and other python objects, making it easy to share them across projects and with your colleagues.
• quilt - Quilt is a data mesh for connecting people with actionable data
• pystore - Fast data store for Pandas time-series data
• pandas - Flexible and powerful data analysis / manipulation library for Python
• pyarrow - Universal columnar format and multi-language toolbox
• parquet - Apache Parquet Format
• fsspec - Filesystem interfaces for Python
• universal_pathlib - pathlib api extended to use fsspec backends