Skip to main content

ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order

Project description

ruamel.yaml

ruamel.yaml is a YAML package for Python. It is a derivative of Kirill Simonov’s PyYAML 3.11 which supports YAML1.1

Major differences with PyYAML 3.11:

  • intergrated Python 2 and 3 sources, running on Python 2.6, 2.7 (CPython, PyPy), 3.3 and 3.4.

  • round trip mode that includes comments (block mode, key ordering kept)

  • support for simple lists as mapping keys by transformation to tuples

  • !!omap generates ordereddict (C) on Python 2, collections.OrderedDict on Python 3, and !!omap is generated for these types.

  • some YAML 1.2 enhancements (0o octal prefix, \/ escape)

  • pep8 compliance

  • tox and py.test based testing

  • Tests whether the C yaml library is installed as well as the header files. That library doesn’t generate CommentTokens, so it cannot be used to do round trip editing on comments. It can be used for speeded up normal processing (so you don’t need to install ruamel.yaml and PyYaml). See the section Optional requirements.

  • Basic support for multiline strings with preserved newlines and chomping ( ‘|’, ‘|+’, ‘|-’ ). As this subclasses the string type the information is lost on reassignment. (This might be changed in the future so that the preservation/folding/chomping is part of the parent container, like comments).

  • RoundTrip preservation of flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)

  • anchors names that are hand-crafted (not of the form``idNNN``), are preserved

  • merges in dictionaries are preserved

  • adding/replacing of comments on block style sequences and mappings with smart column positioning

  • collection objects (when read in via RoundTripParser) have an lc property that contains line and column info lc.line and lc.col

Round trip including comments

The major motivation for this fork is the round-trip capability for comments. The integration of the sources was just an initial step to make this easier.

adding/replacing comments

Starting with version 0.8, you can add/replace comments on block style collections (mappings/sequences resuting in Python dict/list). The basic for for this is:

from __future__ import print_function

import ruamel.yaml

inp = """\
abc:
  - a     # comment 1
xyz:
  a: 1    # comment 2
  b: 2
  c: 3
  d: 4
  e: 5
  f: 6 # comment 3
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
data['abc'].append('b')
data['abc'].yaml_add_eol_comment('comment 4', 1)  # takes column of comment 1
data['xyz'].yaml_add_eol_comment('comment 5', 'c')  # takes column of comment 2
data['xyz'].yaml_add_eol_comment('comment 6', 'e')  # takes column of comment 3
data['xyz'].yaml_add_eol_comment('comment 7', 'd', column=20)

print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in:

abc:
- a       # comment 1
- b       # comment 4
xyz:
  a: 1    # comment 2
  b: 2
  c: 3    # comment 5
  d: 4              # comment 7
  e: 5 # comment 6
  f: 6 # comment 3

If the comment doesn’t start with ‘#’, this will be added. The key is is the element index for list, the actual key for dictionaries. As can be seen from the example, the column to choose for a comment is derived from the previous, next or preceding comment column (picking the first one found).

Config file formats

There are only a few configuration file formats that are easily readable, and editable: JSON, INI/ConfigParser, YAML (XML is to cluttered to be called easily readable).

Unfortunately JSON doesn’t support comments, and although there are some solutions with pre-processed filtering of comments, there are no libraries that support round trip updating of such commented files.

INI files support comments, and the excellent ConfigObj library by Foord and Larosa even supports round trip editing with comment preservation, nesting of sections and limited lists (within a value). Retrieval of particular value format is explicit (and extensible).

YAML has basic mapping and sequence structures as well support for ordered mappings and sets. It supports scalars are of various types including dates and datetimes (missing in JSON) as a list of YAML has comments, but these are normally thrown away.

Block structured YAML is a clean and very human readable format. By extending the Python YAML parser to support round trip preservation of comments, it makes YAML a very good choice for configuration files that are human readable and editable while at the same time interpretable and modifiable by a program.

Extending

There are normally 6 files involved when extending the roundtrip capabilities: the reader, parser, composer and constructor to go from YAML to Python and the resolver, representer, serializer and emitter to go the other way.

Extending involves keeping extra data around for the next process step, eventuallly resulting in a different Python object (subclass or alternative), that should behave like the original, but on the way from Python to YAML generates the original (or at least something much closer).

Smartening

When you use round-tripping, then the complex data you get are already subclasses of the build in types. So you can patch in extra methods or override existing ones. Some methods are already included and you can do:

yaml_str = """\
a:
- b:
  c: 42
- d:
    f: 196
  e:
    g: 3.14
"""


data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

assert data.mlget(['a', 1, 'd', 'f'], list_ok=True) == 196

Examples

Basic round trip of parsing YAML to Python objects, modifying and generating YAML:

from __future__ import print_function

import ruamel.yaml

inp = """\
# example
name:
  # details
  family: Smith   # very common
  given: Alice    # one of the siblings
"""

code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
code['name']['given'] = 'Bob'

print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')

Resulting in

# example
name:
  # details
  family: Smith   # very common
  given: Bob      # one of the siblings

YAML handcrafted anchors and references as well as key merging is preserved. The merged keys can transparently be accessed using [] and .get():

import ruamel.yaml

inp = """\
- &CENTER {x: 1, y: 2}
- &LEFT {x: 0, y: 2}
- &BIG {r: 10}
- &SMALL {r: 1}
# All the following maps are equal:
# Explicit keys
- x: 1
  y: 2
  r: 10
  label: center/big
# Merge one map
- <<: *CENTER
  r: 10
  label: center/big
# Merge multiple maps
- <<: [*CENTER, *BIG]
  label: center/big
# Override
- <<: [*BIG, *LEFT, *SMALL]
  x: 1
  label: center/big
"""

data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
assert data[7]['y'] == 2

Optional requirements

If you have the C yaml library and headers installed, as well as the header files for your Python executables then you can use the non-roundtrip but faster C loader en emitter.

On Debian systems you should use:

sudo apt-get install libyaml-dev python-dev python3-dev

you can leave out python3-dev if you don’t use python3

For CentOS (7) based systems you should do:

sudo yum install libyaml-devel python-devel

Testing

Testing is done using the tox, which uses virtualenv and pytest.

yaml utlity

A utility name yaml is included and allows for basic operations on files:

  • yaml round-trip <file_name> for basic roundtrip testing of YAML files

  • yaml json <file_name> for conversion of JSON file(s) to a single YAML block style document

  • yaml ini <file_name> for conversion of an INI/config file (ConfigObj comment and nested sections supported) to a YAML block style document. This requires configobj to be installed (pip install configobj)

  • yaml from-csv <file_name> for conversion CSV to a YAML file to a a table in an HTML file.

  • yaml htmltable <file_name> for conversion of the basic structure in a YAML file to a a table in an HTML file. The YAML file:

    title:
    - fruit
    - legume
    local:
    - apple
    - sprouts
    import:
    - orange
    - broccoli

    is converted into the table:

    title

    fruit

    legume

    local

    apple

    sprouts

    import

    orange

    broccoli

See yaml --help for more information on the availble commands

Project details


Release history Release notifications | RSS feed

This version

0.10

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ruamel.yaml-0.10.tar.gz (173.7 kB view details)

Uploaded Source

Built Distributions

ruamel.yaml-0.10-cp34-none-win_amd64.whl (74.0 kB view details)

Uploaded CPython 3.4Windows x86-64

ruamel.yaml-0.10-cp34-none-win32.whl (74.0 kB view details)

Uploaded CPython 3.4Windows x86

ruamel.yaml-0.10-cp33-none-win_amd64.whl (74.0 kB view details)

Uploaded CPython 3.3Windows x86-64

ruamel.yaml-0.10-cp33-none-win32.whl (74.0 kB view details)

Uploaded CPython 3.3Windows x86

ruamel.yaml-0.10-cp27-none-win_amd64.whl (74.1 kB view details)

Uploaded CPython 2.7Windows x86-64

ruamel.yaml-0.10-cp27-none-win32.whl (74.1 kB view details)

Uploaded CPython 2.7Windows x86

ruamel.yaml-0.10-cp26-none-win_amd64.whl (74.1 kB view details)

Uploaded CPython 2.6Windows x86-64

ruamel.yaml-0.10-cp26-none-win32.whl (74.1 kB view details)

Uploaded CPython 2.6Windows x86

File details

Details for the file ruamel.yaml-0.10.tar.gz.

File metadata

  • Download URL: ruamel.yaml-0.10.tar.gz
  • Upload date:
  • Size: 173.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for ruamel.yaml-0.10.tar.gz
Algorithm Hash digest
SHA256 b9bcc5232820d12cf7d2a7df277eca6e2dcff40cd4cbedf4a41b5b008a5eb601
MD5 ceea937a1fc1fc5074b880161e4d7eee
BLAKE2b-256 e14c10954ed9cd58f1461bbf330d0327671adc2c976a0024a27beb3c2504fa6f

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp34-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp34-none-win_amd64.whl
Algorithm Hash digest
SHA256 af0656e76607205eb1765c86d4386dff4868395723aad9c6590b5728da7511d9
MD5 a054e8ad923ff1079885b586f4055dbd
BLAKE2b-256 5694b1c951d6496a199204e43ecb28f93aefd2f1b90518c0b99d1a455124c3ee

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp34-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp34-none-win32.whl
Algorithm Hash digest
SHA256 5cc36a63701a015c6317b983dab000b83a69aff1f3e70d0122a19ed914b09bd7
MD5 84259a3ee50818d8f7c1c24c7505b700
BLAKE2b-256 b65a02fad075ea62b051524a5c109762ae1c83d7749eb58e8263c3bf79994410

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp33-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp33-none-win_amd64.whl
Algorithm Hash digest
SHA256 6a84587e2c390e12e192b6179b8a4900fc429754becd0e9473889f963fec1c04
MD5 1ed78480d516abc9ae79911d8ea36a6f
BLAKE2b-256 5ab68d81ed68c91d4de3e0ccb3f2798cace663461dca30f6e515b12d6e108815

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp33-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp33-none-win32.whl
Algorithm Hash digest
SHA256 8bfc859a66d8d5b37e1421bb37f1c0a4bfbc422de8d7cb3680e9d9b80ae2e897
MD5 015065ee636000b55cd58aedad69449b
BLAKE2b-256 5e624bce8ad325029c74bec8037f8e3e59e307072b0ed607f0c84012c6d9083b

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp27-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp27-none-win_amd64.whl
Algorithm Hash digest
SHA256 3cdac321871f358ec7d5a5152ab1ccaf248820e4c6568377b448750c9e203ff0
MD5 577021327f679ed463fce9d0fda1ac60
BLAKE2b-256 b55b0bcb9add5c43e11c8102477a56179fbe1b5f3e7654abb016abd1190400d6

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp27-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp27-none-win32.whl
Algorithm Hash digest
SHA256 b15c1b16fb1221ada525585ab428894da53459a3ad3e23002dd73c8dae435078
MD5 0657f52732888e1bd720bf77dc455207
BLAKE2b-256 852a316e1eaf2aa80431c2b7744bd2768bbd35bfd02b713b37e13c97ec907ab0

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp26-none-win_amd64.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp26-none-win_amd64.whl
Algorithm Hash digest
SHA256 d99b4b818b9afd8856bbbfe2c451205f7373f3493ff3abfdb860c5d0f15e7249
MD5 cc228fc0c793700cfdc2789734395a0b
BLAKE2b-256 d3bd7a10503a2e20d680f8b5c72249ffda7dcabd3b40c7747f43f6e24e1f7d6a

See more details on using hashes here.

File details

Details for the file ruamel.yaml-0.10-cp26-none-win32.whl.

File metadata

File hashes

Hashes for ruamel.yaml-0.10-cp26-none-win32.whl
Algorithm Hash digest
SHA256 697e76d302b4f52b7fe6e07b900ae07aab694f340bb2ae9332b8ca1adcd69975
MD5 3ccd95351b2c8e3fc0c23444cace1339
BLAKE2b-256 3ac561678c77aa3cd6a4de09a69f3202539d2c475ccd54e96dadc75fdd1fc393

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page