litfix

Repair and parse Python/JSON-like literal strings that contain raw, unescaped control characters embedded inside quoted string values.

The problem

>>> import ast
>>> data = "{'title': 'Azur\r\nLane', 'new': True}"
>>> ast.literal_eval(data)
SyntaxError: unterminated string literal (detected at line 1)

This happens when upstream code builds a dict/list containing strings that already have real \r/\n bytes in them (e.g. scraped HTML, multi-line titles), then serializes it with str(obj) instead of repr(obj) or json.dumps(obj). The result looks like a normal Python-repr or JSON string, but has literal control characters sitting inside the quotes -- which is invalid syntax for ast.literal_eval, json.loads, and lenient parsers like json5 alike, since none of them allow a bare newline inside a quoted string.

The fix

litfix walks the source character by character, tracks whether the cursor is inside a quoted string (honoring backslash escapes), and re-escapes control characters only when they're inside quotes. Whitespace between tokens (e.g. pretty-printed line wrapping) is left untouched, since it's already valid syntax there.

>>> from litfix import parse
>>> parse("{'title': 'Azur\r\nLane', 'new': True}")
{'title': 'Azur\r\nLane', 'new': True}

Install

pip install litfix          # once published
pip install -e ".[dev]"     # local editable install with test deps

Zero runtime dependencies.

API

from litfix import (
    parse,                # auto-detect: Python literal, then JSON
    robust_literal_eval,   # ast.literal_eval with auto-repair
    robust_json_loads,     # json.loads with auto-repair
    sanitize_literal,      # just the repair step, for custom pipelines
    RepairReport,          # diagnostics: what got fixed, and where
    LiteralRepairError,    # raised if repair still can't make it parse
)

parse(s, *, verbose=False)

Best-effort parse: tries Python-literal syntax first (single quotes, True/False/None), falls back to JSON. This is the one you want if you don't know or don't care which flavor the source uses.

parse("[{'a': 1, 'b': True}]")        # -> [{'a': 1, 'b': True}]
parse('{"a": 1, "b": true}')          # -> {'a': 1, 'b': True}

robust_literal_eval(s, *, verbose=False) / robust_json_loads(s, *, verbose=False)

Same repair strategy, pinned to one syntax. Pass verbose=True to get back (result, RepairReport) instead of just result:

result, report = robust_literal_eval(raw, verbose=True)
if report.was_modified:
    print(report)   # "repaired 2 control char(s) inside string literals ('\r'x1, '\n'x1)"

sanitize_literal(s, *, report=None)

The raw repair pass, if you want to wire it into your own pipeline before doing something other than literal_eval/json.loads with it.

Errors

If the input still can't be parsed after sanitization, litfix raises LiteralRepairError (a ValueError) with .original_error (the underlying SyntaxError/JSONDecodeError) and .source (the sanitized text that was attempted), so you can see exactly what was tried.

CLI

litfix dump.txt                       # parse -> pretty JSON on stdout
litfix dump.txt -o clean.json         # write to a file instead
cat dump.txt | litfix -v              # read stdin, print repair diagnostics to stderr
litfix dump.txt --mode literal        # force Python-literal parsing
litfix dump.txt --indent 0            # compact single-line JSON output

Exit code is 0 on success, 1 on unrecoverable parse failure (with an error message on stderr).

Testing

pip install -e ".[dev]"
pytest

License

MIT

👤 Author

Hadi Cahyadi

Support me on Patreon