Simple data catalog library for python

This project is a trivial attempt at offering basic catalog functionality for structured datasets stored in local or remote folders. The library uses universal_pathlib to access remote storage locations like S3, Google Cloud Storage, etc ... The library reads a config file called fsdata.ini which defines a list of collections, one per section. Each collection corresponds to a local or remote folder containing data files, homogeneous in format: parquet collections hold DataFrames (pandas or polars), json collections hold plain lists/dicts — declared per collection in the config (format = json; parquet is the default). Consumers pick the accessor for the shape they want (.pandas(), .polars(), .json()); the wrong accessor for the collection format raises. The library uses local caching to avoid fetching the same data multiple times.

Warning This project is for exploration only, the interface can change.

Configuration

The configuration file fsdata.ini has one section for each collection, with the section name for name and with a path key pointing to its location. The file is resolved in order: the FSDATA_CONFIG environment variable, then upward search from the current directory (a repo-local config), then the standard XDG config directory XDG_CONFIG_HOME (or ~/.config). The first match wins; configs never merge.

Each collection declares its format with a format key (parquet or json); parquet is the default when the key is omitted.

Values support strict ${VAR} environment-variable interpolation, so a committed repo-local config can declare the collection layout while the storage root stays in the environment (e.g. provisioned by direnv). Unset variables raise a clear error on access — never a silent literal path.

# fsdata.ini

[samples]
path = ${MY_LAKE}/samples

[datasets]
path = ${MY_LAKE}/datasets
format = parquet

[tickers]
path = ${MY_LAKE}/tickers
format = json

To assert that the active config defines the collections your code depends on, fail fast with fsdata.require("tickers", "datasets"), or from the shell:

python -m fsdata check            # validate all configured collections
python -m fsdata check tickers    # validate specific collections

Usage

To access a collection use the collection function, or — for collection names that are valid identifiers — plain attribute access on the module.

import fsdata

samples = fsdata.collection("samples")
samples = fsdata.samples                  # same thing

To list the configured collections

fsdata.collection_names()

To list items in a collection (item names are bare names, without extension)

samples.items()
samples.has("my-sample")

To load data, pick the accessor for the shape you want. Each accessor has a single concrete return type, and raises if the collection format does not match.

samples.pandas("my-sample")     # -> pandas.DataFrame   (parquet collections)
samples.polars("my-sample")     # -> polars.DataFrame   (parquet collections)

tickers = fsdata.tickers
tickers.json("DOW30")           # -> plain list or dict (json collections)

To save data use the save method — the object type must match the collection format: pandas/polars DataFrames go to parquet collections, plain lists and dicts go to json collections. Anything else raises.

samples.save("my-sample", df)          # DataFrame -> .parquet
tickers.save("DOW30", ["MMM", "AXP"])  # list -> .json
samples.remove("my-sample")            # delete an item

To inspect a parquet item without loading it, use metadata — it reads only the parquet footer, so it stays fast on large files. For the raw artifact bytes, use read_bytes.

meta = samples.metadata("my-sample")
meta.num_rows, meta.schema, meta.num_row_groups

raw = samples.read_bytes("my-sample")

Caching

Remote collections (s3, gs, az) keep a local copy of each item they read, so repeated loads do not re-fetch. Local collections are read in place and never cached.

Cached files live in $XDG_CACHE_HOME/fsdata/<collection>/ (usually ~/.cache/fsdata/<collection>/), one file per item. The directory is created on first use, and deleting it is always safe.

A cached item is reused without contacting storage for check_interval seconds (one day by default). After that, fsdata compares timestamps and re-fetches only if the stored artifact actually changed. Saving or removing an item drops its cached copy immediately.

samples = fsdata.collection("samples")
samples.pandas("my-sample", refresh=True)   # ignore the cache, re-fetch now

Set the window per collection when constructing one directly — check_interval=0 checks storage on every access.

from fsdata import Collection

samples = Collection("samples", "s3://my-bucket/samples", check_interval=0)

Deprecated APIs

The following functions still work but emit a DeprecationWarning; new code should use the replacements.

The backend= parameter is superseded by the accessor names: instead of selecting the return type with an argument, call the accessor that returns what you want.

Installation

You can install the package with pip

pip install fsdata

The frame backends are extras — install the one(s) matching the accessors you use: pandas, polars. Cloud storage backends are extras as well: s3, gcs, adl install the required fsspec backend.

pip install "fsdata[pandas,s3]"

Requirements

• pandas and/or polars (each needed only by its own accessor — see extras)
• pyarrow
• universal_pathlib
• fsspec backends like s3fs, etc ... as applicable (see extras)

Related Projects and Resources

• intake - Lightweight package for finding, investigating, loading and disseminating data.
• pins - Publish data sets, models, and other python objects, making it easy to share them across projects and with your colleagues.
• quilt - Quilt is a data mesh for connecting people with actionable data
• pystore - Fast data store for Pandas time-series data
• pandas - Flexible and powerful data analysis / manipulation library for Python
• pyarrow - Universal columnar format and multi-language toolbox
• parquet - Apache Parquet Format
• fsspec - Filesystem interfaces for Python
• universal_pathlib - pathlib api extended to use fsspec backends