Skip to main content

ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order

Project description

ruamel.yaml

ruamel.yaml is a YAML package for Python. It is a derivative of Kirill Simonov’s PyYAML 3.11 which supports YAML1.1

Major differences with PyYAML 3.11:

  • intergrated Python 2 and 3 sources, running on Python 2.6, 2.7 (CPython, PyPy), 3.3 and 3.4.

  • round trip mode that includes comments (block mode, key ordering kept)

  • support for simple lists as mapping keys by transformation to tuples

  • !!omap generates ordereddict (C) on Python 2, collections.OrderedDict on Python 3, and !!omap is generated for these types.

  • some YAML 1.2 enhancements (0o octal prefix, \/ escape)

  • pep8 compliance

  • tox and py.test based testing

  • Tests whether the C yaml library is installed as well as the header files. That library doesn’t generate CommentTokens, so it cannot be used to do round trip editing on comments. It can be used for speeded up normal processing (so you don’t need to install ruamel.yaml and PyYaml). See the section Optional requirements.

  • Basic support for multiline strings with preserved newlines and chomping ( ‘|’, ‘|+’, ‘|-’ ). As this subclasses the string type the information is lost on reassignment. (This might be changed in the future so that the preservation/folding/chomping is part of the parent container, like comments).

  • RoundTrip preservation of flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)

  • anchors names that are hand-crafted (not of the form``idNNN``), are preserved

  • merges in dictionaries are preserved

  • adding/replacing of comments on block style sequences and mappings with smart column positioning

  • collection objects (when read in via RoundTripParser) have an lc property that contains line and column info lc.line and lc.col

Round trip including comments

The major motivation for this fork is the round-trip capability for comments. The integration of the sources was just an initial step to make this easier.

adding/replacing comments

Starting with version 0.8, you can add/replace comments on block style collections (mappings/sequences resuting in Python dict/list). The basic for for this is:

from __future__ import print_function

import ruamel.yaml

inp = """\
abc:
  - a     # comment 1
xyz:
  a: 1    # comment 2
  b: 2
  c: 3
  d: 4
  e: 5
  f: 6 # comment 3
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
data['abc'].append('b')
data['abc'].yaml_add_eol_comment('comment 4', 1)  # takes column of comment 1
data['xyz'].yaml_add_eol_comment('comment 5', 'c')  # takes column of comment 2
data['xyz'].yaml_add_eol_comment('comment 6', 'e')  # takes column of comment 3
data['xyz'].yaml_add_eol_comment('comment 7', 'd', column=20)

print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in:

abc:
- a       # comment 1
- b       # comment 4
xyz:
  a: 1    # comment 2
  b: 2
  c: 3    # comment 5
  d: 4              # comment 7
  e: 5 # comment 6
  f: 6 # comment 3

If the comment doesn’t start with ‘#’, this will be added. The key is is the element index for list, the actual key for dictionaries. As can be seen from the example, the column to choose for a comment is derived from the previous, next or preceding comment column (picking the first one found).

Config file formats

There are only a few configuration file formats that are easily readable, and editable: JSON, INI/ConfigParser, YAML (XML is to cluttered to be called easily readable).

Unfortunately JSON doesn’t support comments, and although there are some solutions with pre-processed filtering of comments, there are no libraries that support round trip updating of such commented files.

INI files support comments, and the excellent ConfigObj library by Foord and Larosa even supports round trip editing with comment preservation, nesting of sections and limited lists (within a value). Retrieval of particular value format is explicit (and extensible).

YAML has basic mapping and sequence structures as well support for ordered mappings and sets. It supports scalars are of various types including dates and datetimes (missing in JSON) as a list of YAML has comments, but these are normally thrown away.

Block structured YAML is a clean and very human readable format. By extending the Python YAML parser to support round trip preservation of comments, it makes YAML a very good choice for configuration files that are human readable and editable while at the same time interpretable and modifiable by a program.

Extending

There are normally 6 files involved when extending the roundtrip capabilities: the reader, parser, composer and constructor to go from YAML to Python and the resolver, representer, serializer and emitter to go the other way.

Extending involves keeping extra data around for the next process step, eventuallly resulting in a different Python object (subclass or alternative), that should behave like the original, but on the way from Python to YAML generates the original (or at least something much closer).

Smartening

When you use round-tripping, then the complex data you get are already subclasses of the build in types. So you can patch in extra methods or override existing ones. Some methods are already included and you can do:

yaml_str = """\
a:
- b:
  c: 42
- d:
    f: 196
  e:
    g: 3.14
"""


data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

assert data.mlget(['a', 1, 'd', 'f'], list_ok=True) == 196

Examples

Basic round trip of parsing YAML to Python objects, modifying and generating YAML:

from __future__ import print_function

import ruamel.yaml

inp = """\
# example
name:
  # details
  family: Smith   # very common
  given: Alice    # one of the siblings
"""

code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
code['name']['given'] = 'Bob'

print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in

# example
name:
  # details
  family: Smith   # very common
  given: Bob      # one of the siblings

YAML handcrafted anchors and references as well as key merging is preserved. The merged keys can transparently be accessed using [] and .get():

import ruamel.yaml

inp = """\
- &CENTER {x: 1, y: 2}
- &LEFT {x: 0, y: 2}
- &BIG {r: 10}
- &SMALL {r: 1}
# All the following maps are equal:
# Explicit keys
- x: 1
  y: 2
  r: 10
  label: center/big
# Merge one map
- <<: *CENTER
  r: 10
  label: center/big
# Merge multiple maps
- <<: [*CENTER, *BIG]
  label: center/big
# Override
- <<: [*BIG, *LEFT, *SMALL]
  x: 1
  label: center/big
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
assert data[7]['y'] == 2

Optional requirements

If you have the C yaml library and headers installed, as well as the header files for your Python executables then you can use the non-roundtrip but faster C loader en emitter.

On Debian systems you should use:

sudo apt-get install libyaml-dev python-dev python3-dev

you can leave out python3-dev if you don’t use python3

For CentOS (7) based systems you should do:

sudo yum install libyaml-devel python-devel

Testing

Testing is done using the tox, which uses virtualenv and pytest.

yaml utlity

A utility name yaml is included and allows for basic operations on files:

  • yaml round-trip <file_name> for basic roundtrip testing of YAML files

  • yaml json <file_name> for conversion of JSON file(s) to a single YAML block style document

  • yaml ini <file_name> for conversion of an INI/config file (ConfigObj comment and nested sections supported) to a YAML block style document. This requires configobj to be installed (pip install configobj)

  • yaml from-csv <file_name> for conversion CSV to a YAML file to a a table in an HTML file.

  • yaml htmltable <file_name> for conversion of the basic structure in a YAML file to a a table in an HTML file. The YAML file:

    title:
    - fruit
    - legume
    local:
    - apple
    - sprouts
    import:
    - orange
    - broccoli

    is converted into the table:

    title

    fruit

    legume

    local

    apple

    sprouts

    import

    orange

    broccoli

See yaml --help for more information on the availble commands

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ruamel.yaml-0.10.2.tar.gz (174.7 kB view details)

Uploaded Source

Built Distributions

ruamel.yaml-0.10.2-cp34-none-win_amd64.whl (74.4 kB view details)

Uploaded CPython 3.4Windows x86-64

ruamel.yaml-0.10.2-cp34-none-win32.whl (74.4 kB view details)

Uploaded CPython 3.4Windows x86

ruamel.yaml-0.10.2-cp33-none-win_amd64.whl (74.4 kB view details)

Uploaded CPython 3.3Windows x86-64

ruamel.yaml-0.10.2-cp33-none-win32.whl (74.4 kB view details)

Uploaded CPython 3.3Windows x86

ruamel.yaml-0.10.2-cp27-none-win_amd64.whl (74.5 kB view details)

Uploaded CPython 2.7Windows x86-64

ruamel.yaml-0.10.2-cp27-none-win32.whl (145.1 kB view details)

Uploaded CPython 2.7Windows x86

ruamel.yaml-0.10.2-cp26-none-win_amd64.whl (74.5 kB view details)

Uploaded CPython 2.6Windows x86-64

ruamel.yaml-0.10.2-cp26-none-win32.whl (74.5 kB view details)

Uploaded CPython 2.6Windows x86

File details

Details for the file ruamel.yaml-0.10.2.tar.gz.

File metadata

  • Download URL: ruamel.yaml-0.10.2.tar.gz
  • Upload date:
  • Size: 174.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for ruamel.yaml-0.10.2.tar.gz
Algorithm Hash digest
SHA256 27e28bdb6c9bba0db99fe4cb8b55d3aaaa699b13f11970a1bb81e37cf769d725
MD5 75970cceaa604df39fa22d869364e04d
BLAKE2b-256 e1a726c09d5c63c9e9d1837751c7e17fafb2ab4c8753a35d19fde64d902bafcf

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp34-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp34-none-win_amd64.whl
Algorithm Hash digest
SHA256 d3fe0c150a0cc144b8cbc33d490845b5ff796733c9afbada936a2f636fd9613e
MD5 f9716bed5b23755694f7c38a8bd458f0
BLAKE2b-256 a2ae3b8b9667632d1120190bd3b094927825d58e70fd4f5643a0e4a6d5b96ceb

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp34-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp34-none-win32.whl
Algorithm Hash digest
SHA256 91586d8e3ed268f182a651ba02dd9342ca89d10df344bdb3ab52d315568a19ca
MD5 febe2f87486ad9b63d8dcd56c3e3c41c
BLAKE2b-256 92db9498fae511fc4442ac99d6f388c5f57b26d3dad8b48c8af113703e4526a6

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp33-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp33-none-win_amd64.whl
Algorithm Hash digest
SHA256 722d494ad946ed68a7d1a4ebaa6040851b7f5a960241247d0c3ec0eb08cd0647
MD5 93e27b0e9aca13c8b98aa61f2f54d5bd
BLAKE2b-256 414d203a05b789e719a5fd730e38a05857b2756bd3d82929e02be84314b20e30

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp33-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp33-none-win32.whl
Algorithm Hash digest
SHA256 0a6073a7d49c7ffb0ab7c629978fc427fa8b14abc7649ca84414b772f3e308b7
MD5 867a9bc86c6ef97ad0b03402c6836d8e
BLAKE2b-256 d836d6f71fdb46f4ef5cd05dc2359e7e5455d8a049ebc321138f0c173f4e712d

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp27-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp27-none-win_amd64.whl
Algorithm Hash digest
SHA256 3b0f0b802986631e71f50904cd8324a7fde1563fe4498badbb3330a6aeef3c82
MD5 cd9ca5d7c1c94312674f6a62c4bd28d7
BLAKE2b-256 3b0e7107dc33c03fe48db69b9d9eff5ab7559c72946fd033e369c04fdb8dfaf6

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp27-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp27-none-win32.whl
Algorithm Hash digest
SHA256 756d7d94110a46a783d897ab9d423ed0ade7fea05fed0f6d646718b16cdbf637
MD5 212cdc6ba05f4a2084b5f5f9c12f7b10
BLAKE2b-256 3ee8628e5a83e20377290553abfafb0ec8ae8fa1c8c8f7e133f903d55cfbcf35

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp26-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp26-none-win_amd64.whl
Algorithm Hash digest
SHA256 17b6fce4e43773a950062174d58872e6dd89a3a1154bcf7ffc52954ef1633ab6
MD5 9d5359bf65f8a7816f28dcd3d165c68f
BLAKE2b-256 891480b409f58b2d1a65758ace9722514cc9d6de0833b000f11eed7895d1ba3c

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10.2-cp26-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10.2-cp26-none-win32.whl
Algorithm Hash digest
SHA256 1db1373b6886a53896748c2075ff37df3997a0a7ee69121068c36692b58ff384
MD5 8f13f722f9fdb3267e01695c43415118
BLAKE2b-256 b01d5ea14723c8205a9d0de6787b40628153d504dcb83dda4795bf50f55ebf97

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page