fschema

Marshmallow-like schematization of a directory structure

Installation

Simply install it from PyPI

pip install fschema

Quickstart Example

Let's say you have the following directory/file structure:

config/
  + plugins/
  |   + java/
  |   |   + plugin.yaml
  |   + python/
  |   |   + plugin.yaml
  + profiles/
  |   + new.yaml
  |   + old.yaml
  + env
  + config.yaml

You can describe it as a Python model and load everything into a single structure. Schemas describe the structure, fields describe how each node is loaded, and filesystem access goes through an FSInterface. FSLoader uses LocalFSInterface by default, but custom filesystem-like backends can be provided with FSLoader(schema=..., fs=...).

from fschema.fields import meta, node
from fschema.schema import Schema
from fschema.fs_loader import FSLoader

class PluginConfigSchema(Schema):
    name = meta.NodeName()
    config = node.File(fs_name="plugin.yaml")

class ProfileConfigSchema(Schema):
    name = meta.NodeName()
    config = meta.Content()

class ServiceConfigSchema(Schema):
    config = node.File(fs_name="config.yaml")
    env = node.File(fs_name="env")
    plugins = node.ListDirectory(node.SchematizedDirectory(PluginConfigSchema()))
    profiles = node.ListDirectory(node.SchematizedFile(ProfileConfigSchema()))

data = FSLoader(schema=ServiceConfigSchema()).load("/path/to/config")
print(data)

This will load the following data:

{
  "config": "<file-content>",
  "env": "<file-content>",
  "plugins": [
    {"name": "java", "config":  "<file-content>"},
    {"name": "python", "config":  "<file-content>"}
  ],
  "profiles": [
    {"name": "new.yaml", "config":  "<file-content>"},
    {"name": "old.yaml", "config":  "<file-content>"}
  ]
}

If you want to add post-processing of the data to your schema (e.g. validate it or convert it to an object), you can define a __fschema_post_load__ method:

class ServiceConfigSchema(Schema):
    ...
    def __fschema_post_load__(self, data: dict) -> ServiceConfiguration:
        return ServiceConfiguration(**data)

Reference

Fields

Meta Fields

Meta fields are the fields that use the metadata of the respective filesystem node (directory/file) and provide access to its various properties. All meta fields inherit from MetaField.

Meta field types:

• NodeName() - special type of field that loads the name of the current node (directory or file)
• Content(reader: Reader, data_transformer: DataTransformer) - for use inside a sub-schema of a SchematizedFile; reader parses content provided by FSLoader to JSON-like data; data_transformer loads it into an object and/or validates the data

Node Fields

Node fields correspond to actual filesystem nodes (directories/fields). All node fields inherit from NodeField.

All node fields have the optional argument fs_name - this is the name of the filesystem node the field corresponds to - useful if the filename has a period (.) in it, and, therefore cannot be used as the field's attribute name.

Exposed properties:

• effective_fs_name - the resolved filesystem name: the explicit fs_name when provided, otherwise the schema attribute name.

Node field types:

• SchematizedDirectory(directory_schema: Schema, fs_name: str | None) - load directory as a key-value mapping and apply the given sub-schema to the directory itself; this means nested files and directories must have fixed names
• DictDirectory(nested_field: Field, fs_name: str | None) - load directory as a free mapping, without fixed key values; the given field instance is applied to all nested nodes
• ListDirectory(nested_field: Field, fs_name: str | None) - load directory as a list of nodes; the given field instance is applied to all nested nodes
• File(fs_name: str | None, reader: Reader, data_transformer: DataTransformer) - load file content; reader parses content provided by FSLoader to JSON-like data; data_transformer loads it into an object and/or validates the data
• SchematizedFile(file_schema: Schema, fs_name: str | None) - load the file as a schematized mapping instead of a single flat object; this is useful if you need access to its metadata (e.g. via NodeName);

Content Readers

Available content readers:

• JSONReader - parses content as JSON (as a dict)
• YamlReader - parses content as YAML (as a dict)
• TextReader - returns content as text (str); this is the default reader

Filesystem Interface

Filesystem access is abstracted behind FSInterface, available from fschema.fs. The default LocalFSInterface, also available from fschema.fs, supports local paths via pathlib. Custom backends can implement:

• node_name(path: Path) -> str
• child_path(path: Path, fs_name: str) -> Path
• list_directory(path: Path) -> list[Path]
• require_file(path: Path) -> None
• require_directory(path: Path) -> None
• read_file(path: Path, encoding="utf-8") -> str

Data Transformers

Available data transformers:

• MarshmallowLoader(schema: Any) - loads the file data via a marshmallow schema

Extending

If you need custom behavior, the main extension points are:

• fschema.fields.meta.MetaField - base class for fields that read the current filesystem node
• fschema.fields.node.NodeField - base class for fields that resolve/load filesystem child nodes
• fschema.fs.FSInterface - base class for filesystem-like backends
• fschema.readers.Reader - base class for content readers
• fschema.data.DataTransformer - base class for data transformers

See the corresponding source modules for the expected method shapes and examples.

Contribute

Feel free to contribute. I can't guarantee I'll review PRs fast, but I'll do my best.

Setup

For local development, create a virtual environment and install the development extras:

python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"

If your tools live somewhere else, copy .make.env.example to .make.env and adjust the paths:

cp .make.env.example .make.env

The local .make.env file is ignored by git and can contain machine-specific values:

PYTHON = venv-fschema-3.14/bin/python
RUFF = venv-fschema-3.14/bin/ruff

Checking

Before opening a pull request, run:

make test

make test first runs make format-check, then runs the unit tests. To fix lint and formatting issues automatically, run:

make format

You can also run the validation step directly:

make format-check