Skip to main content

ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order

Project description

ruamel.yaml

ruamel.yaml is a YAML package for Python. It is a derivative of Kirill Simonov’s PyYAML 3.11 which supports YAML1.1

Major differences with PyYAML 3.11:

  • intergrated Python 2 and 3 sources, running on Python 2.6, 2.7 (CPython, PyPy), 3.3 and 3.4.

  • round trip mode that includes comments (block mode, key ordering kept)

  • support for simple lists as mapping keys by transformation to tuples

  • !!omap generates ordereddict (C) on Python 2, collections.OrderedDict on Python 3, and !!omap is generated for these types.

  • some YAML 1.2 enhancements (0o octal prefix, \/ escape)

  • pep8 compliance

  • tox and py.test based testing

  • Tests whether the C yaml library is installed as well as the header files. That library doesn’t generate CommentTokens, so it cannot be used to do round trip editing on comments. It can be used for speeded up normal processing (so you don’t need to install ruamel.yaml and PyYaml). See the section Optional requirements.

  • Basic support for multiline strings with preserved newlines and chomping ( ‘|’, ‘|+’, ‘|-’ ). As this subclasses the string type the information is lost on reassignment. (This might be changed in the future so that the preservation/folding/chomping is part of the parent container, like comments).

  • RoundTrip preservation of flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)

  • anchors names that are hand-crafted (not of the form``idNNN``), are preserved

  • merges in dictionaries are preserved

  • adding/replacing of comments on block style sequences and mappings with smart column positioning

  • collection objects (when read in via RoundTripParser) have an lc property that contains line and column info lc.line and lc.col

Round trip including comments

The major motivation for this fork is the round-trip capability for comments. The integration of the sources was just an initial step to make this easier.

adding/replacing comments

Starting with version 0.8, you can add/replace comments on block style collections (mappings/sequences resuting in Python dict/list). The basic for for this is:

from __future__ import print_function

import ruamel.yaml

inp = """\
abc:
  - a     # comment 1
xyz:
  a: 1    # comment 2
  b: 2
  c: 3
  d: 4
  e: 5
  f: 6 # comment 3
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
data['abc'].append('b')
data['abc'].yaml_add_eol_comment('comment 4', 1)  # takes column of comment 1
data['xyz'].yaml_add_eol_comment('comment 5', 'c')  # takes column of comment 2
data['xyz'].yaml_add_eol_comment('comment 6', 'e')  # takes column of comment 3
data['xyz'].yaml_add_eol_comment('comment 7', 'd', column=20)

print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in:

abc:
- a       # comment 1
- b       # comment 4
xyz:
  a: 1    # comment 2
  b: 2
  c: 3    # comment 5
  d: 4              # comment 7
  e: 5 # comment 6
  f: 6 # comment 3

If the comment doesn’t start with ‘#’, this will be added. The key is is the element index for list, the actual key for dictionaries. As can be seen from the example, the column to choose for a comment is derived from the previous, next or preceding comment column (picking the first one found).

Config file formats

There are only a few configuration file formats that are easily readable, and editable: JSON, INI/ConfigParser, YAML (XML is to cluttered to be called easily readable).

Unfortunately JSON doesn’t support comments, and although there are some solutions with pre-processed filtering of comments, there are no libraries that support round trip updating of such commented files.

INI files support comments, and the excellent ConfigObj library by Foord and Larosa even supports round trip editing with comment preservation, nesting of sections and limited lists (within a value). Retrieval of particular value format is explicit (and extensible).

YAML has basic mapping and sequence structures as well support for ordered mappings and sets. It supports scalars are of various types including dates and datetimes (missing in JSON) as a list of YAML has comments, but these are normally thrown away.

Block structured YAML is a clean and very human readable format. By extending the Python YAML parser to support round trip preservation of comments, it makes YAML a very good choice for configuration files that are human readable and editable while at the same time interpretable and modifiable by a program.

Extending

There are normally 6 files involved when extending the roundtrip capabilities: the reader, parser, composer and constructor to go from YAML to Python and the resolver, representer, serializer and emitter to go the other way.

Extending involves keeping extra data around for the next process step, eventuallly resulting in a different Python object (subclass or alternative), that should behave like the original, but on the way from Python to YAML generates the original (or at least something much closer).

Smartening

When you use round-tripping, then the complex data you get are already subclasses of the build in types. So you can patch in extra methods or override existing ones. Some methods are already included and you can do:

yaml_str = """\
a:
- b:
  c: 42
- d:
    f: 196
  e:
    g: 3.14
"""


data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

assert data.mlget(['a', 1, 'd', 'f'], list_ok=True) == 196

Examples

Basic round trip of parsing YAML to Python objects, modifying and generating YAML:

from __future__ import print_function

import ruamel.yaml

inp = """\
# example
name:
  # details
  family: Smith   # very common
  given: Alice    # one of the siblings
"""

code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
code['name']['given'] = 'Bob'

print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in

# example
name:
  # details
  family: Smith   # very common
  given: Bob      # one of the siblings

YAML handcrafted anchors and references as well as key merging is preserved. The merged keys can transparently be accessed using [] and .get():

import ruamel.yaml

inp = """\
- &CENTER {x: 1, y: 2}
- &LEFT {x: 0, y: 2}
- &BIG {r: 10}
- &SMALL {r: 1}
# All the following maps are equal:
# Explicit keys
- x: 1
  y: 2
  r: 10
  label: center/big
# Merge one map
- <<: *CENTER
  r: 10
  label: center/big
# Merge multiple maps
- <<: [*CENTER, *BIG]
  label: center/big
# Override
- <<: [*BIG, *LEFT, *SMALL]
  x: 1
  label: center/big
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
assert data[7]['y'] == 2

Optional requirements

If you have the C yaml library and headers installed, as well as the header files for your Python executables then you can use the non-roundtrip but faster C loader en emitter.

On Debian systems you should use:

sudo apt-get install libyaml-dev python-dev python3-dev

you can leave out python3-dev if you don’t use python3

For CentOS (7) based systems you should do:

sudo yum install libyaml-devel python-devel

Testing

Testing is done using the tox, which uses virtualenv and pytest.

yaml utlity

A utility name yaml is included and allows for basic operations on files:

  • yaml round-trip <file_name> for basic roundtrip testing of YAML files

  • yaml json <file_name> for conversion of JSON file(s) to a single YAML block style document

  • yaml ini <file_name> for conversion of an INI/config file (ConfigObj comment and nested sections supported) to a YAML block style document. This requires configobj to be installed (pip install configobj)

  • yaml from-csv <file_name> for conversion CSV to a YAML file to a a table in an HTML file.

  • yaml htmltable <file_name> for conversion of the basic structure in a YAML file to a a table in an HTML file. The YAML file:

    title:
    - fruit
    - legume
    local:
    - apple
    - sprouts
    import:
    - orange
    - broccoli

    is converted into the table:

    title

    fruit

    legume

    local

    apple

    sprouts

    import

    orange

    broccoli

See yaml --help for more information on the availble commands

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ruamel.yaml-0.10.1.tar.gz (174.6 kB view details)

Uploaded Source

Built Distributions

ruamel.yaml-0.10.1-cp34-none-win_amd64.whl (74.4 kB view details)

Uploaded CPython 3.4Windows x86-64

ruamel.yaml-0.10.1-cp34-none-win32.whl (74.4 kB view details)

Uploaded CPython 3.4Windows x86

ruamel.yaml-0.10.1-cp33-none-win_amd64.whl (74.4 kB view details)

Uploaded CPython 3.3Windows x86-64

ruamel.yaml-0.10.1-cp33-none-win32.whl (74.4 kB view details)

Uploaded CPython 3.3Windows x86

ruamel.yaml-0.10.1-cp27-none-win_amd64.whl (74.5 kB view details)

Uploaded CPython 2.7Windows x86-64

ruamel.yaml-0.10.1-cp27-none-win32.whl (144.7 kB view details)

Uploaded CPython 2.7Windows x86

ruamel.yaml-0.10.1-cp26-none-win_amd64.whl (74.5 kB view details)

Uploaded CPython 2.6Windows x86-64

ruamel.yaml-0.10.1-cp26-none-win32.whl (74.5 kB view details)

Uploaded CPython 2.6Windows x86

File details

Details for the file ruamel.yaml-0.10.1.tar.gz.

File metadata

  • Download URL: ruamel.yaml-0.10.1.tar.gz
  • Upload date:
  • Size: 174.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for ruamel.yaml-0.10.1.tar.gz
Algorithm Hash digest
SHA256 b3a87f6d0763f59d3845069136e2867c7a475887b4f3db0f773cc31117ff001b
MD5 3ed1522d4aa1f2e59522588454b14a7f
BLAKE2b-256 5c96e9a7d221980c5d1267f48c19412bb6f113d953d9e473611beca04ffda29b

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp34-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp34-none-win_amd64.whl
Algorithm Hash digest
SHA256 9f67ebfbc7ba17e1051e631756ec36da773fe7451983e2a27353a42010e18530
MD5 091c0d32d944ca0aefd48a67d755ab6c
BLAKE2b-256 6a2d1e4432a19641510a4d1c5ceeba3334c832aebe606e1a51896a2ac62bab35

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp34-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp34-none-win32.whl
Algorithm Hash digest
SHA256 a9d841fafb7696c4b9aa6bd81b003000b733e5fbe3f859576157597355ad5c66
MD5 5df6b1b284ec1bcb9003f757d49ec7df
BLAKE2b-256 cc351a2563498ebd022725174529437a0394d54d230a76c69369739d4b9c6568

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp33-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp33-none-win_amd64.whl
Algorithm Hash digest
SHA256 8581398f07766952f083d8abe6e537100782b2405ddd7fd36e75adb4434b6062
MD5 dab1523685b5ed255ce8311e197010f5
BLAKE2b-256 c36d7cfc9ec49a58c5a1bab1b7610fbe3de4edefc837023b46826c767af0d3f1

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp33-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp33-none-win32.whl
Algorithm Hash digest
SHA256 0258be2008d09a5790f82c1423aa9db3edd39a2208d27f4d62a54675add0f8af
MD5 bb338973e8bba66f6acdb3362a2df40b
BLAKE2b-256 deb4a14019dec184be7cc445d8b1542f041d663c75069e5161845ba8611d447c

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp27-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp27-none-win_amd64.whl
Algorithm Hash digest
SHA256 b2525a79ff5fa7ccb492927cb05473b0dd0f2189d23aeba1c01343283181fcf0
MD5 62537d8c30cd2b308bb5ca2a066b1e6f
BLAKE2b-256 f6d34a2cd2a04f65f4e26f3920901981f25a4509514b04a2da5a07d3e98f06d7

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp27-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp27-none-win32.whl
Algorithm Hash digest
SHA256 70109d6e6c636be99cf322c37e88c0574a1f0f91cb075ba49ef1ddeb34176961
MD5 8d3514858ef61d343cd041492b285ff9
BLAKE2b-256 c0a0eef2f43cc233feafdcc863538b15dd8208eb19fee9f54a95d87544ffd494

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp26-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp26-none-win_amd64.whl
Algorithm Hash digest
SHA256 7d7f60543e391018df5a82931217cd8a4b7df5e422f023cfb15ea04a174fa2f0
MD5 241e6b9bf74d9a652ce273e7285b2d1f
BLAKE2b-256 a6258ec856b0a416469e2ff8dee10d7be8b44d449990d737af9d818c8f33076a

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.1-cp26-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.1-cp26-none-win32.whl
Algorithm Hash digest
SHA256 4e60899e47c9d48839d7deca167c5977fd5c06a2a1868b652a7f2c1fb0df5ea1
MD5 8b93485b6eecc75607b8145ad084c1a8
BLAKE2b-256 676ac146b914ff9cc94dc6cdd93b940ec24c97a63f557577d5658869a75395df

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page