litfix
Repair and parse Python/JSON-like literal strings that contain raw, unescaped control characters embedded inside quoted string values.
The problem
>>> import ast
>>> data = "{'title': 'Azur\r\nLane', 'new': True}"
>>> ast.literal_eval(data)
SyntaxError: unterminated string literal (detected at line 1)
This happens when upstream code builds a dict/list containing strings
that already have real \r/\n bytes in them (e.g. scraped HTML,
multi-line titles), then serializes it with str(obj) instead of
repr(obj) or json.dumps(obj). The result looks like a normal
Python-repr or JSON string, but has literal control characters sitting
inside the quotes -- which is invalid syntax for ast.literal_eval,
json.loads, and lenient parsers like json5 alike, since none of them
allow a bare newline inside a quoted string.
The fix
litfix walks the source character by character, tracks whether the
cursor is inside a quoted string (honoring backslash escapes), and
re-escapes control characters only when they're inside quotes.
Whitespace between tokens (e.g. pretty-printed line wrapping) is left
untouched, since it's already valid syntax there.
>>> from litfix import parse
>>> parse("{'title': 'Azur\r\nLane', 'new': True}")
{'title': 'Azur\r\nLane', 'new': True}
Install
pip install litfix # once published
pip install -e ".[dev]" # local editable install with test deps
Zero runtime dependencies.
API
from litfix import (
parse, # auto-detect: Python literal, then JSON
robust_literal_eval, # ast.literal_eval with auto-repair
robust_json_loads, # json.loads with auto-repair
sanitize_literal, # just the repair step, for custom pipelines
RepairReport, # diagnostics: what got fixed, and where
LiteralRepairError, # raised if repair still can't make it parse
)
parse(s, *, verbose=False)
Best-effort parse: tries Python-literal syntax first (single quotes,
True/False/None), falls back to JSON. This is the one you want if
you don't know or don't care which flavor the source uses.
parse("[{'a': 1, 'b': True}]") # -> [{'a': 1, 'b': True}]
parse('{"a": 1, "b": true}') # -> {'a': 1, 'b': True}
robust_literal_eval(s, *, verbose=False) / robust_json_loads(s, *, verbose=False)
Same repair strategy, pinned to one syntax. Pass verbose=True to get
back (result, RepairReport) instead of just result:
result, report = robust_literal_eval(raw, verbose=True)
if report.was_modified:
print(report) # "repaired 2 control char(s) inside string literals ('\r'x1, '\n'x1)"
sanitize_literal(s, *, report=None)
The raw repair pass, if you want to wire it into your own pipeline
before doing something other than literal_eval/json.loads with it.
Errors
If the input still can't be parsed after sanitization, litfix raises
LiteralRepairError (a ValueError) with .original_error (the
underlying SyntaxError/JSONDecodeError) and .source (the sanitized
text that was attempted), so you can see exactly what was tried.
CLI
litfix dump.txt # parse -> pretty JSON on stdout
litfix dump.txt -o clean.json # write to a file instead
cat dump.txt | litfix -v # read stdin, print repair diagnostics to stderr
litfix dump.txt --mode literal # force Python-literal parsing
litfix dump.txt --indent 0 # compact single-line JSON output
Exit code is 0 on success, 1 on unrecoverable parse failure (with an
error message on stderr).
Testing
pip install -e ".[dev]"
pytest
License
MIT
👤 Author
Metadata
Release files for litfix 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| litfix-1.0.0.tar.gz | 10.6 kB | Without provenance | Details for litfix-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| litfix-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for litfix-1.0.0-py3-none-any.whl |
Total release size: 20.4 kB
Release files / litfix-1.0.0.tar.gz
| Download URL | litfix-1.0.0.tar.gz |
|---|---|
| Size | 10.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e2c14788209b1a9efe5f2df6c9d9dfff315410a88bea5fbb4cab6fa72090161b
|
|
BLAKE2b-256 checksum How to use checksums |
917bc8b14282b5aed9b128ffb3aef077c95e234c51a398970b09361ff3c258bb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Release files / litfix-1.0.0-py3-none-any.whl
| Download URL | litfix-1.0.0-py3-none-any.whl |
|---|---|
| Size | 9.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
95c2b27a9bf7129dcc116de2601501c9930209cad54be7d80669dcee0f1c8d12
|
|
BLAKE2b-256 checksum How to use checksums |
39c5a9cd5e1a1ac69608dc8a9f21f79218746298533bb20c4a65bbb1573e71a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Security for litfix 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.