py-jsonl 1.3.0

A simple Python library for handling jsonlines files.

jsonl

About

Useful functions for working with jsonlines data as described: https://jsonlines.org/

Features:

• Offers an API similar to Python's built-in json module.
• Supports serialization/deserialization using the most common json libraries, prioritizing orjson, then ujson, and defaulting to the standard json if the others are unavailable.
• Enables compression using gzip, bzip2, and xz formats.

Installation (via pip)

pip install py-jsonl

Usage

Serialize an iterable into a JSON Lines formatted string. (dumps)

Examples:

import jsonl

data = ({'foo': 1}, {'bar': 2})
result = jsonl.dumps(data)
print(result)

Dump an iterable to a JSON Lines file. (dump)

Examples:

Write the data to an uncompressed file at the specified path.

import jsonl

data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

jsonl.dump(data, "file.jsonl")  # as list
jsonl.dump(iter(data), "file.jsonl")  # as iterable

Write the data to a compressed file at the specified path.

import jsonl

data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

jsonl.dump(data, "file.jsonl.gz")
jsonl.dump(data, "file.jsonl.bz2")
jsonl.dump(data, "file.jsonl.xz")

Write the data to the already opened gzipped file.

import gzip
import jsonl

data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

with gzip.open("file.jsonl.gz", mode="wb") as fp:
    jsonl.dump(data, fp, text_mode=False)

Append the data to the end of the existing gzipped file.

import gzip
import jsonl

data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

with gzip.open("file.jsonl.gz", mode="ab") as fp:
    jsonl.dump(data, fp, text_mode=False)

Dump fork (Incremental dump)

Incrementally dumps multiple iterables into the specified jsonlines file paths, effectively reducing memory consumption.

Examples:

import jsonl

path_iterables = (
    ("num.jsonl", ({"value": 1}, {"value": 2})),
    ("foo.jsonl", ({"a": "1"}, {"b": 2})),
    ("num.jsonl", ({"value": 3},)),
    ("foo.jsonl", ()),
)
jsonl.dump_fork(path_iterables)

load

Deserialize a UTF-8 encoded jsonlines file into an iterable of Python objects.

Examples:

Load an uncompressed file from the specified path.

import jsonl

path = "file.jsonl"
data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

jsonl.dump(data, path)
iterable = jsonl.load(path)
print(tuple(iterable))

Load a compressed file from the specified path.

import jsonl

path = "file.jsonl.gz"
data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

jsonl.dump(data, path)
iterable = jsonl.load(path)
print(tuple(iterable))

Load a compressed file from the specified open file object.

import gzip
import jsonl

path = "file.jsonl.gz"
data = [
    {"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]},
    {"name": "May", "wins": []},
]

jsonl.dump(data, path)
with gzip.open(path, mode="rb") as fp:
    iterable = jsonl.load(fp)
    print(tuple(iterable))

Unit tests

(env)$ pip install -r requirements.txt   # Ignore this command if it has already been executed
(env)$ pytest tests/
(env)$ pytest --cov jsonl # Tests with coverge