Skip to main content

ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order

Project description

ruamel.yaml

Starting with 0.11.0 the RoundTripLoader differentiates between YAML 1.2 and YAML 1.1. This may cause compatibility problems, see the “Document version support” section below.


Starting with 0.10.7 the package has been reorganised and the command line utility is in its own package ruamel.yaml.cmd (so installing ruamel.yaml doesn’t pull in possibly irrelevant modules only used in the command line utility)

ruamel.yaml is a YAML package for Python. It is a derivative of Kirill Simonov’s PyYAML 3.11 which supports YAML1.1

Major differences with PyYAML 3.11:

  • integrated Python 2 and 3 sources, running on Python 2.6, 2.7 (CPython, PyPy), 3.3 and 3.4.

  • round trip mode that includes comments (block mode, key ordering kept)

  • support for simple lists as mapping keys by transforming these to tuples

  • !!omap generates ordereddict (C) on Python 2, collections.OrderedDict on Python 3, and !!omap is generated for these types.

  • some YAML 1.2 enhancements (0o octal prefix, \/ escape)

  • pep8 compliance

  • tox and py.test based testing

  • Tests whether the C yaml library is installed as well as the header files. That library doesn’t generate CommentTokens, so it cannot be used to do round trip editing on comments. It can be used to speed up normal processing (so you don’t need to install ruamel.yaml and PyYaml). See the section Optional requirements.

  • Basic support for multiline strings with preserved newlines and chomping ( ‘|’, ‘|+’, ‘|-’ ). As this subclasses the string type the information is lost on reassignment. (This might be changed in the future so that the preservation/folding/chomping is part of the parent container, like comments).

  • RoundTrip preservation of flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)

  • anchors names that are hand-crafted (not of the form``idNNN``) are preserved

  • merges in dictionaries are preserved

  • adding/replacing comments on block-style sequences and mappings with smart column positioning

  • collection objects (when read in via RoundTripParser) have an lc property that contains line and column info lc.line and lc.col. Individual positions for mappings and sequences can also be retrieved (lc.key('a'), lc.value('a') resp. lc.item(3))

  • preservation of whitelines after block scalars. Contributed by Sam Thursfield.

Document version support.

In YAML a document version can be explicitly set by using:

%YAML 1.x

before the document start (at the top or before a ---). For ruamel.yaml x has to be 1 or 2. If no explicit version is set version 1.2 is assumed (which has been released in 2009).

The 1.2 version does not support:

  • sexagesimals like 12:34:56

  • octals that start with 0 only: like 012 for number 10 (0o12 is supported by YAML 1.2)

  • Unquoted Yes and On as alternatives for True and No and Off for False.

If you cannot change your YAML files and you need them to load as 1.1 you can load with:

ruamel.yaml.load(some_str, Loader=ruamel.yaml.RoundTripLoader, version=(1, 1))

or the equivalent (version can be a tuple, list or string):

ruamel.yaml.round_trip_load(some_str, version=”1.1”)

this also works for load_all/round_trip_load_all.

If you cannot change your code, stick with ruamel.yaml==0.10.23 and let me know if it would help to be able to set an environment variable.

This does not affect dump as ruamel.yaml never emitted sexagesimals, nor octal numbers, and emitted booleans always as true resp. false

Round trip including comments

The major motivation for this fork is the round-trip capability for comments. The integration of the sources was just an initial step to make this easier.

adding/replacing comments

Starting with version 0.8, you can add/replace comments on block style collections (mappings/sequences resuting in Python dict/list). The basic for for this is:

from __future__ import print_function

import ruamel.yaml

inp = """\
abc:
  - a     # comment 1
xyz:
  a: 1    # comment 2
  b: 2
  c: 3
  d: 4
  e: 5
  f: 6 # comment 3
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
data['abc'].append('b')
data['abc'].yaml_add_eol_comment('comment 4', 1)  # takes column of comment 1
data['xyz'].yaml_add_eol_comment('comment 5', 'c')  # takes column of comment 2
data['xyz'].yaml_add_eol_comment('comment 6', 'e')  # takes column of comment 3
data['xyz'].yaml_add_eol_comment('comment 7', 'd', column=20)

print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in:

abc:
- a       # comment 1
- b       # comment 4
xyz:
  a: 1    # comment 2
  b: 2
  c: 3    # comment 5
  d: 4              # comment 7
  e: 5 # comment 6
  f: 6 # comment 3

If the comment doesn’t start with ‘#’, this will be added. The key is the element index for list, the actual key for dictionaries. As can be seen from the example, the column to choose for a comment is derived from the previous, next or preceding comment column (picking the first one found).

Config file formats

There are only a few configuration file formats that are easily readable and editable: JSON, INI/ConfigParser, YAML (XML is to cluttered to be called easily readable).

Unfortunately JSON doesn’t support comments, and although there are some solutions with pre-processed filtering of comments, there are no libraries that support round trip updating of such commented files.

INI files support comments, and the excellent ConfigObj library by Foord and Larosa even supports round trip editing with comment preservation, nesting of sections and limited lists (within a value). Retrieval of particular value format is explicit (and extensible).

YAML has basic mapping and sequence structures as well as support for ordered mappings and sets. It supports scalars various types including dates and datetimes (missing in JSON). YAML has comments, but these are normally thrown away.

Block structured YAML is a clean and very human readable format. By extending the Python YAML parser to support round trip preservation of comments, it makes YAML a very good choice for configuration files that are human readable and editable while at the same time interpretable and modifiable by a program.

Extending

There are normally six files involved when extending the roundtrip capabilities: the reader, parser, composer and constructor to go from YAML to Python and the resolver, representer, serializer and emitter to go the other way.

Extending involves keeping extra data around for the next process step, eventuallly resulting in a different Python object (subclass or alternative), that should behave like the original, but on the way from Python to YAML generates the original (or at least something much closer).

Smartening

When you use round-tripping, then the complex data you get are already subclasses of the built-in types. So you can patch in extra methods or override existing ones. Some methods are already included and you can do:

yaml_str = """\
a:
- b:
  c: 42
- d:
    f: 196
  e:
    g: 3.14
"""


data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

assert data.mlget(['a', 1, 'd', 'f'], list_ok=True) == 196

Examples

Basic round trip of parsing YAML to Python objects, modifying and generating YAML:

from __future__ import print_function

import ruamel.yaml

inp = """\
# example
name:
  # details
  family: Smith   # very common
  given: Alice    # one of the siblings
"""

code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
code['name']['given'] = 'Bob'

print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in

# example
name:
  # details
  family: Smith   # very common
  given: Bob      # one of the siblings

YAML handcrafted anchors and references as well as key merging is preserved. The merged keys can transparently be accessed using [] and .get():

import ruamel.yaml

inp = """\
- &CENTER {x: 1, y: 2}
- &LEFT {x: 0, y: 2}
- &BIG {r: 10}
- &SMALL {r: 1}
# All the following maps are equal:
# Explicit keys
- x: 1
  y: 2
  r: 10
  label: center/big
# Merge one map
- <<: *CENTER
  r: 10
  label: center/big
# Merge multiple maps
- <<: [*CENTER, *BIG]
  label: center/big
# Override
- <<: [*BIG, *LEFT, *SMALL]
  x: 1
  label: center/big
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
assert data[7]['y'] == 2

Optional requirements

If you have the C yaml library and headers installed, as well as the header files for your Python executables then you can use the non-roundtrip but faster C loader and emitter.

On Debian systems you should use:

sudo apt-get install libyaml-dev python-dev python3-dev

you can leave out python3-dev if you don’t use python3

For CentOS (7) based systems you should do:

sudo yum install libyaml-devel python-devel

Testing

Testing is done using tox, which uses virtualenv and pytest.

ChangeLog

0.11.0 (2016-02-18):
  - RoundTripLoader loads 1.2 by default (no sexagesimals, 012 octals nor
    yes/no/on/off booleans

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ruamel.yaml-0.11.0.tar.gz (231.6 kB view details)

Uploaded Source

Built Distributions

ruamel.yaml-0.11.0-py3-none-win_amd64.whl (72.3 kB view details)

Uploaded Python 3Windows x86-64

ruamel.yaml-0.11.0-py3-none-win32.whl (72.3 kB view details)

Uploaded Python 3Windows x86

ruamel.yaml-0.11.0-py2-none-win_amd64.whl (72.3 kB view details)

Uploaded Python 2Windows x86-64

ruamel.yaml-0.11.0-py2-none-win32.whl (72.3 kB view details)

Uploaded Python 2Windows x86

File details

Details for the file ruamel.yaml-0.11.0.tar.gz.

File metadata

  • Download URL: ruamel.yaml-0.11.0.tar.gz
  • Upload date:
  • Size: 231.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for ruamel.yaml-0.11.0.tar.gz
Algorithm Hash digest
SHA256 e55b5547d89dba782206489d2fbdcff2d0d7048272cf254eaf8af008193d0ee1
MD5 0edeabebb429c7209e8184c701368770
BLAKE2b-256 ff8f15600e8a437667471afabc29f26ca95242cd65fdeb98fef1e239bd1c86c9

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.11.0-py3-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.11.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 68c25cedf7edfe78673529af66e65034051a5316d30aec4ea4e48712c6bb267f
MD5 7b109c9894168903ee49cf23f8c27fa0
BLAKE2b-256 7490aa93dcea9b7c187720fa19aa2d8ebf3b5e07dee054df0093bee3a336ec77

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.11.0-py3-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.11.0-py3-none-win32.whl
Algorithm Hash digest
SHA256 60aa4829eb2983638410107fcf6e7909514a509295446c7aaabf8f71df42fdca
MD5 75ee8dba387ec6b28d798eb090f694fa
BLAKE2b-256 db3e56fe78dda1c9a80823ecd3f8fc6c07f41faeb79d7cc549f2dcdc87f9d174

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.11.0-py2-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.11.0-py2-none-win_amd64.whl
Algorithm Hash digest
SHA256 87cc0af56c1b2b53286709736ef903a78c0cb2f98f4a92ed134f73fac599811c
MD5 ed6763338a1767db94481c2c66b273ff
BLAKE2b-256 62daa82d2e1c5eef048e11c1890730b4f31c5d38bff7274e21afbda3121a862b

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.11.0-py2-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.11.0-py2-none-win32.whl
Algorithm Hash digest
SHA256 a1e7326cf1498a1244ada2a81802e2fc3ad09887978b662d4882721c13b59db8
MD5 bc8640f661121fddd5a4eec1354e88ab
BLAKE2b-256 89b3124ea7afa79c7b8ee3c9b8fe9d30443607e857210a6069cbb524765aa197

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page