cs-ndjson 20250528

utilities for working with newline delimited JSON (NDJSON) files

Utilities for working with newline delimited JSON (NDJSON) files, aka JSON Lines.

Latest release 20250528: New dump_ndjson and dumps_ndjson for transcribing obj as NDJSON (includes the trailing newline).

Short summary:

• append_ndjson: Append an iterable of objects to a file as newline delimited JSON.
• dump_ndjson: Write obj as a single line of JSON, including the trailing newline.
• dumps_ndjson: Return obj as a single line of JSON, including the trailing newline.
• scan_ndjson: Read a newline delimited JSON file, yield instances of dictclass (default dict, otherwise a class which can be instantiated by dictclass(a_dict)).
• UUIDNDJSONMapping: A subclass of IndexedSetMixin which maintains records from a newline delimited JSON file.
• write_ndjson: Transcribe an iterable of objects to a file as newline delimited JSON.

Module contents:

• append_ndjson(f, objs): Append an iterable of objects to a file as newline delimited JSON.

• dump_ndjson(obj, f): Write obj as a single line of JSON, including the trailing newline.

• dumps_ndjson(obj): Return obj as a single line of JSON, including the trailing newline.

• scan_ndjson(f, dictclass=<class 'dict'>, error_list=None): Read a newline delimited JSON file, yield instances of dictclass (default dict, otherwise a class which can be instantiated by dictclass(a_dict)).

  error_list is an optional list to accrue (lineno,exception) tuples for errors encountered during the scan.

• Class UUIDNDJSONMapping(cs.obj.SingletonMixin, cs.mappings.IndexedSetMixin)``: A subclass of IndexedSetMixin which maintains records from a newline delimited JSON file.

UUIDNDJSONMapping.__init__(self, filename, dictclass=<class 'cs.mappings.UUIDedDict'>, create=False): Initialise the mapping.

Parameters:

• filename: the file containing the newline delimited JSON data; this need not yet exist
• dictclass: a optional dict subclass to hold each record, default UUIDedDict
• create: if true, ensure the file exists by transiently opening it for append if it is missing; default False

UUIDNDJSONMapping.add_backend(self, record): Append record to the backing file.

UUIDNDJSONMapping.rewrite_backend(self): Rewrite the backing file.

Because the record updates are normally written in append mode, a rewrite will be required every so often.

UUIDNDJSONMapping.scan(self): Scan the backing file, yield records.

• write_ndjson(f, objs): Transcribe an iterable of objects to a file as newline delimited JSON.

Release Log

Release 20250528: New dump_ndjson and dumps_ndjson for transcribing obj as NDJSON (includes the trailing newline).

Release 20211208: Initial PyPI release.