yaml-extras

Some extra scripting utilities written on top of PyYAML.

See the full docs at yaml-extras.pages.dev.

Installation

Install with pip or poetry:

pip install yaml-extras
# or
poetry add yaml-extras

Usage

import yaml
from yaml_extras import ExtrasLoader
import json

with open('example.yml') as f:
    data = yaml.load(f, Loader=ExtrasLoader)

print(f"data = {json.dumps(data, indent=2)}")

Features

Modularity with "import"

!import tag: Import another whole YAML file. Supports multiple-import using the "<<" merge key, as well as aliasing the result of an import using an anchor.

Note: There is no safeguard against cyclical imports. If you import a file that imports the original file, it will result in exceeding Python's maximum recursion depth.

Syntax

!import [&anchor ]<filepath>

Examples

Simple whole-file import

# example.yml
my_children:
  child1: !import child1.yml
  child2: !import child2.yml

# child1.yml
name: child1
age: 10

# child2.yml
name: child2
age: 7

Result when loading in Python:

data = {
  "my_children": {
    "child1": {
      "name": "child1",
      "age": 10
    },
    "child2": {
      "name": "child2",
      "age": 7
    }
  }
}

Nested whole-file imports

# example.yml
my_children:
  child: !import child.yml

# child.yml
name: child
age: 40
grandchild: !import grandchild.yml

# grandchild.yml
name: grandchild
age: 10

Result when loading in Python:

data = {
  "my_children": {
    "child": {
      "name": "child",
      "age": 40,
      "grandchild": {
        "name": "grandchild",
        "age": 10
      }
    }
  }
}

Multiple whole-file imports with merge

# example.yml
all_animals:
  <<: 
    - !import animals/american.yml
    - !import animals/eurasian.yml

# animals/american.yml
bear: wild
wolf: wild
fish: domestic
dog: domestic

# animals/eurasian.yml
tiger: wild
lion: wild
cat: domestic
chicken: domestic

Result when loading in Python:

data = {
  "all_animals": {
    "bear": "wild",
    "wolf": "wild",
    "fish": "domestic",
    "dog": "domestic",
    "tiger": "wild",
    "lion": "wild",
    "cat": "domestic",
    "chicken": "domestic"
  }
}

Anchored whole-file imports

# example.yml
child: !import &child-anchor child.yml
again:
  child: *child-anchor

# child.yml
name: child
age: 10

Result when loading in Python:

data = {
  "child": {
    "name": "child",
    "age": 10
  },
  "again": {
    "child": {
      "name": "child",
      "age": 10
    }
  }
}

!import.anchor tag: Import a specific anchor from another YAML file. Supports multiple-import using the "<<" merge key, as well as aliasing the result of an import using an anchor.

Note: There is no safeguard against cyclical imports. If you import an anchor that imports the original file (or anchor you define with this import), it will result in exceeding Python's maximum recursion depth.

Syntax

!import.anchor [&internal_anchor ]<filepath> &<external_anchor>

Examples

Simple anchor import

# example.yml
my_children:
  child1: !import.anchor children.yml &child1
  child2: !import.anchor children.yml &child2

# children.yml
child1: &child1
  name: child1
  age: 10
child2: &child2
  name: child2
  age: 7

Result when loading in Python:

data = {
  "my_children": {
    "child1": {
      "name": "child1",
      "age": 10
    },
    "child2": {
      "name": "child2",
      "age": 7
    }
  }
}

Nested anchor imports

# example.yml
my_children:
  child: !import.anchor child.yml &child

# child.yml
name: child
age: 40
grandchild: !import.anchor grandchild.yml &grandchild

# grandchild.yml
grandchild: &grandchild
  name: grandchild
  age: 10

Result when loading in Python:

data = {
  "my_children": {
    "child": {
      "name": "child",
      "age": 40,
      "grandchild": {
        "name": "grandchild",
        "age": 10
      }
    }
  }
}

Multiple anchor imports with merge

# example.yml
all_animals:
  <<: 
    - !import.anchor animals.yml &american
    - !import.anchor animals.yml &eurasian

# animals.yml
american: &american
  bear: wild
  wolf: wild
  fish: domestic
  dog: domestic

eurasian: &eurasian
  tiger: wild
  lion: wild
  cat: domestic
  chicken: domestic

Result when loading in Python:

data = {
  "all_animals": {
    "bear": "wild",
    "wolf": "wild",
    "fish": "domestic",
    "dog": "domestic",
    "tiger": "wild",
    "lion": "wild",
    "cat": "domestic",
    "chicken": "domestic"
  }
}

Anchored anchor imports

# example.yml
child: !import.anchor &my-child child.yml &child-anchor
again:
  child: *my-child

# child.yml
child: &child-anchor
  name: child
  age: 10

Result when loading in Python:

data = {
  "child": {
    "name": "child",
    "age": 10
  },
  "again": {
    "child": {
      "name": "child",
      "age": 10
    }
  }
}

!import-all tag: Import a glob pattern of YAML files as a sequence. Supports merging the imports using the "<<" merge key, as well as aliasing the result of an import using an anchor.

The glob pattern system only supports two types of wildcards: * and **. * matches any character except for /, while ** matches any character including /.

Note: There is no safeguard against cyclical imports. If you import a file that imports the original file, it will result in exceeding Python's maximum recursion depth.

Syntax

!import-all [&anchor ]<glob_pattern>

Examples

Simple (*) sequence import

# example.yml
all_children: !import-all children/*.yml

# children/alice.yml
name: alice
age: 10
height: 1.2

#children/bob.yml
name: bob
age: 7
height: 1.0

Result when loading in Python:

data = {
  "all_children": [
    {
      "name": "alice",
      "age": 10,
      "height": 1.2
    },
    {
      "name": "bob",
      "age": 7,
      "height": 1.0
    }
  ]
}

Nested (*) sequence imports

# example.yml
all_children: !import-all children/*.yml

# children/bob.yml
name: bob
age: 28

# children/alice.yml
name: alice
age: 40
children: !import-all children/alice/*.yml

# children/alice/fred.yml
name: fred
age: 10

# children/alice/jane.yml
name: jane
age: 7

Result when loading in Python:

data = {
  "all_children": [
    {
      "name": "bob",
      "age": 28
    },
    {
      "name": "alice",
      "age": 40,
      "children": [
        {
          "name": "fred",
          "age": 10
        },
        {
          "name": "jane",
          "age": 7
        }
      ]
    }
  ]
}

Multiple (*) sequence imports with merge

# example.yml
all_animals:
  <<: 
    - !import-all animals/american/*.yml
    - !import-all animals/eurasian/*.yml

# animals/american/bear.yml
bear:
  type: wild
  size: large

# animals/american/wolf.yml
wolf:
  type: wild
  size: medium

# animals/eurasian/tiger.yml
tiger:
  type: wild
  size: large

# animals/eurasian/lion.yml
lion:
  type: wild
  size: large

Result when loading in Python:

data = {
  "all_animals": {
    "bear": {
      "type": "wild",
      "size": "large"
    },
    "wolf": {
      "type": "wild",
      "size": "medium"
    },
    "tiger": {
      "type": "wild",
      "size": "large"
    },
    "lion": {
      "type": "wild",
      "size": "large"
    }
  }
}

Anchored (*) sequence imports

# example.yml
data: !import-all &my-children children/*.yml
again:
  data: *my-children

# children/alice.yml
name: alice
age: 10
height: 1.2

# children/bob.yml
name: bob
age: 7
height: 1.0

Result when loading in Python:

data = {
  "data": [
    {
      "name": "alice",
      "age": 10,
      "height": 1.2
    },
    {
      "name": "bob",
      "age": 7,
      "height": 1.0
    }
  ],
  "again": {
    "data": [
      {
        "name": "alice",
        "age": 10,
        "height": 1.2
      },
      {
        "name": "bob",
        "age": 7,
        "height": 1.0
      }
    ]
  }
}

Simple (**) sequence import

# example.yml
overarching: !import-all subfolders/**/*.yml

# subfolders/child1.yml
name: child1

# subfolders/child2.yml
name: child2

# subfolders/subfolder1/grandchild1.yml
name: grandchild1

Result when loading in Python:

data = {
  "overarching": [
    {
      "name": "child1"
    },
    {
      "name": "child2"
    },
    {
      "name": "grandchild1"
    }
  ]
}

!import-all-parameterized tag: Import a glob pattern of YAML files as a sequence, enriching the results with one or more metadata keys extracted from globs in the filepath. Supports merging the imports using the "<<" merge key, as well as aliasing the result of an import using an anchor.

The glob pattern system only supports two types of wildcards: * and **. * matches any character except for /, while ** matches any character including /. Metadata keys can be extracted from zero or more globs in the path specification with syntax like this:

# Import all YAML files in the `path/to/*.yml` glob as a sequence, attaching to each element the
# `basename` key extracted from the filename.
my_data: !import-all-parameterized path/to/{basename:*}.yml
#
# my_data:
#  - basename: file1
#    key1: value1
#  - basename: file2
#    key2: value2
#    key3: value3
# 

# Import all YAML files in the `path/to/**/meta.yml` glob as a sequence, attaching to each
# element the `subdirs` key extracted from the subdirectory structure.
my_subdirs: !import-all-parameterized path/to/{subdirs:**}/meta.yml
#
# my_subdirs:
#  - subdirs: subdir1
#    key1: value1
#  - subdirs: subdir1/subdir2/subdir3
#    key2: value2
#    key3: value3
#

Note (i): There is no safeguard against cyclical imports. If you import a file that imports the original file, it will result in exceeding Python's maximum recursion depth.

Note (ii): When the leaf files of an import contain mappings, then it is simple to "merge" the metadata keys from the path into the resulting imported mappings. However, when the leaf files are scalars or sequences, then the structure of the import results are slightly more contrived. The contents of the imports will be under a content key in each result, with the metadata keys extracted from the path added as additional key/value pairs in the mappings.

Syntax

!import-all-parameterized [&anchor ]<glob_pattern>

Examples

Simple parameterized import (*) with metadata

# example.yml
grade_book: !import-all schools/{school_name:*}/grades/{student_name:*}.yml

# schools/elementary/grades/David.yml
math: 95
science: 90
english: 80

# schools/elementary/grades/Edward.yml
math: 100
science: 90
english: 100

# schools/highschool/grades/Frank.yml
math: 85
science: 95
english: 90

Result when loading in Python:

data = {
  "grade_book": [
    {
      "school_name": "elementary",
      "student_name": "David",
      "math": 95,
      "science": 90,
      "english": 80
    },
    {
      "school_name": "elementary",
      "student_name": "Edward",
      "math": 100,
      "science": 90,
      "english": 100
    },
    {
      "school_name": "highschool",
      "student_name": "Frank",
      "math": 85,
      "science": 95,
      "english": 90
    }
  ]
}

Simple parameterized import (**) with metadata

# example.yml
translations: !import-all-parameterized words/{langspec:**}/words.yml

# words/en/us/words.yml
- hello
- goodbye
- color
- thanks

# words/en/uk/words.yml
- good morrow
- toodle-oo
- colour
- cheers

# words/es/mx/words.yml
- hola
- adios
- color
- gracias

Result when loading in Python:

data = {
  "translations": [
    {
      "langspec": "en/us",
      "content": ["hello", "goodbye", "color", "thanks"]
    },
    {
      "langspec": "en/uk",
      "content": ["good morrow", "toodle-oo", "colour", "cheers"]
    },
    {
      "langspec": "es/mx",
      "content": ["hola", "adios", "color", "gracias"]
    }
  ]
}

Customizing the import directory

By default, !import tags will search relative to the current working directory of the Python process. You can customize the base directory for imports by calling yaml_import.set_import_relative_dir(...) with the desired base directory.

import yaml
from yaml_extras import ExtrasLoader, yaml_import

yaml_import.set_import_relative_dir('/path/to/imports')
data = yaml.load('!import somefile.yml', Loader=ExtrasLoader)

Roadmap

P1

Add support for !import to import other whole documents into a YAML document (general import).
Add support for !import.anchor to import specific anchors from other YAML documents (targeted import).
Add support for !import-all to import a glob pattern of YAML files as a sequence.
Add support for !import-all.anchor to import a specific anchor from a glob pattern of YAML files as a sequence.
Add support for !import-all-parameterized to import a glob pattern of YAML files as a sequence with some data extracted from the filepath.
Add support for !import-all-parameterized.anchor to import a specific anchor from a glob pattern of YAML files as a sequence with some data extracted from the filepath.
Allow user to set relative import directory.

P2

Implement type specification system to validate YAML files against a schema using a !yamlschema tag system which mimics JSON Schema semantics and are validated upon construction.
Add support for !env tag to import environment variables.

P3

VSCode / Intellisense plugin to navigate through imports using cmd + click

Acknowledgements

Release files for yaml-extras 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for yaml-extras 0.2.2
File	Size	Uploaded
yaml_extras-0.2.2.tar.gz	14.4 kB	Jan 8, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for yaml-extras 0.2.2
File	Interpreter	ABI	Platform	Reset
yaml_extras-0.2.2-py3-none-any.whl 12.6 kB Jan 8, 2025	Python 3	none	any	Details

Total release size:26.9 kB

Release files / yaml_extras-0.2.2.tar.gz

Download URL	yaml_extras-0.2.2.tar.gz
Size	14.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`61bda0a376a34f398f39818d51d3e85a1ea3ba88c5e43122e075d29971d75dd1`
BLAKE2b-256 checksum How to use checksums	`d025d401f4e6cea220d451b5998c6a0a82d8bc698c07af5fab330a33196d90a1`
Upload date	Jan 8, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.0.1 CPython/3.12.8`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/dsillman2000/yaml-extras
Publishing commit: github.com/dsillman2000/yaml-extras/tree/ed65538437c9af2a0abf6f28e237e5b6bfd997a5

Workflow

Publishing configuration: github.com/dsillman2000/yaml-extras/blob/ed65538437c9af2a0abf6f28e237e5b6bfd997a5/.github/workflows/python-publish.yml
Publishing logs: github.com/dsillman2000/yaml-extras/actions/runs/12674548098/attempts/1

Artifact

Subject: yaml_extras-0.2.2.tar.gz
SHA-256 checksum: 61bda0a376a34f398f39818d51d3e85a1ea3ba88c5e43122e075d29971d75dd1 Verifying attestations

Release files / yaml_extras-0.2.2-py3-none-any.whl

Download URL	yaml_extras-0.2.2-py3-none-any.whl
Size	12.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`59443fbed416841c07281c73882d2bdb2737004d7e892057a743f3ea18725bf9`
BLAKE2b-256 checksum How to use checksums	`3b34ad825f67ae4e1a5ea7c5b9a94003b275cee3de3453ce4b60c97a719f7988`
Upload date	Jan 8, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.0.1 CPython/3.12.8`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/dsillman2000/yaml-extras
Publishing commit: github.com/dsillman2000/yaml-extras/tree/ed65538437c9af2a0abf6f28e237e5b6bfd997a5

Workflow

Publishing configuration: github.com/dsillman2000/yaml-extras/blob/ed65538437c9af2a0abf6f28e237e5b6bfd997a5/.github/workflows/python-publish.yml
Publishing logs: github.com/dsillman2000/yaml-extras/actions/runs/12674548098/attempts/1

Artifact

Subject: yaml_extras-0.2.2-py3-none-any.whl
SHA-256 checksum: 59443fbed416841c07281c73882d2bdb2737004d7e892057a743f3ea18725bf9 Verifying attestations