jsonhelper

JSON utilities for Python, with two guarantees the standard json module does not give you by default: no silent data loss and atomic writes to disk.

Works on Python 2.7 and 3.x from a single codebase, with no dependencies.

import jsonhelper

data = jsonhelper.load("config.json")      # file -> Python object
jsonhelper.save(data, "config.json")       # Python object -> file (atomic)

jsonhelper.loads('{"a": 1}')               # text -> Python object
jsonhelper.dumps({"a": 1})                 # Python object -> text
jsonhelper.pretty('{"a":1}')               # text -> indented text
jsonhelper.validate('{"a": 1}')            # -> True / False

dumps, pretty and save also accept indent= (default 1) and sort_keys= (default False).

What it does differently

The first three are extensions or behaviours that the json module accepts but RFC 8259 does not allow. They produce files that other parsers (JavaScript, Go, Rust) reject, or lose data without any warning at all.

Errors

Every function reports failure by raising:

• ValueError — the text is not valid JSON, or the object is not representable.
• TypeError — an argument of the wrong type was passed.
• EnvironmentError — disk access problems.

About that last one: on Python 3, EnvironmentError is an alias of OSError, so you can catch FileNotFoundError and friends as usual. On Python 2, IOError and OSError are separate classes and IOError does not inherit from OSError; their common base is EnvironmentError. If your code has to run on both, catch EnvironmentError.

try:
    config = jsonhelper.load("config.json")
except EnvironmentError:          # OSError on Python 3
    config = {}
except ValueError as err:
    raise SystemExit("config.json is corrupt: %s" % err)

Atomic writes

save() serialises and validates before touching the disk, writes to a temporary file in the same directory, calls fsync, and then publishes the result with an atomic rename. If anything goes wrong — unserialisable content, a full disk, an interrupted process, another process reading at the same time — the destination file keeps its previous contents intact. There is never a window in which a reader sees the file empty or half-written.

If the path is a symbolic link, the link itself is replaced, not its target.

Python 3 uses os.replace. Python 2 falls back to os.rename, which is already atomic on POSIX, and to MoveFileEx with MOVEFILE_REPLACE_EXISTING on Windows, where os.rename fails if the destination exists.

Reproducible output

Python 2.7 dictionaries do not preserve insertion order, while Python 3.7+ dictionaries do. The same data may therefore be written with its keys in a different order depending on the interpreter. The content is equivalent, but the files are not byte-for-byte identical.

If you commit these files to version control, diff them, or checksum them, pass sort_keys=True — the output is then identical on any version:

jsonhelper.save(data, "config.json", sort_keys=True)
jsonhelper.dumps(data, sort_keys=True)

Installation

pip install jsonhelper

Requires Python 2.7 or 3.3+. No dependencies.

Python 2.7 has been unsupported since 1 January 2020. Compatibility is kept for legacy codebases; use Python 3 for anything new.

Migrating from 0.0.x

Version 0.1.0 is a breaking change. The reason is that the old API returned False on error — a value indistinguishable from the perfectly valid JSON documents false, null, 0, [], {} and "". Callers had no way to tell whether a call had failed.

Things worth checking when you upgrade:

• load() returns data, not text. The old open() returned formatted JSON; load() returns the Python object. For the text, use jsonhelper.dumps(jsonhelper.load(f)).
• save() takes an object, not text. If you already have JSON text: save(loads(text), f).
• open() was renamed to load(), because it shadowed the open builtin.
• Errors are no longer swallowed. Wrap calls in try/except where you used to check if not result:.
• Output is not ASCII-escaped. Accented and CJK characters are written as-is in UTF-8 rather than as \uXXXX. It is still valid JSON.

jsontools is kept as an alias for the module functions, so from jsonhelper import jsontools still works — but the method names are the new ones.

Development

The test suite has no dependencies and runs on the standard library alone:

python    -m unittest discover -s tests   # Python 3
python2.7 -m unittest discover -s tests   # Python 2.7

pytest will also collect it if you prefer. It covers one regression per defect documented in the QA report included in this repository, and is verified on both versions.

License

GPL-3.0-or-later. See LICENSE.txt.