json-streams 0.3.1

Stream objects from json-arrays or json lines lazily..

json-streams

Read and write JSON lazy, especially json-arrays.

Handles both the JSON format:

[
    {
        "a": 1
    },
    {
        "a": 2
    }
]

As well as JSON LINES format:

{"a":1}
{"a": 2}

Uses orjson or ujson if present, otherwise standard json.

Usage

Installation

# Using standard json
pip install json-streams

# Using orjson
pip install json-streams[orjson]

# Using ujson
pip install json-streams[ujson]

Note

This library prefers files opened in binary mode. Therefore does all dumps-methods return bytes.

All loads methods handles str argument. If you use the orjson library you can also pass bytes or bytesarray to loads. The goal is to have all loads handling str, bytes and bytesarray.

Examples

Allows you to use json.load and json.dump with both json and json-lines files as well as dumping generators.

import json_streams

# This command tries to guess format and opens the file
data = json_streams.load_from_file("data.json") # or data.jsonl

# Write to file, again guessing format
json_streams.dump_to_file(data, "data.jsonl")

from json_streams import json_iter, jsonl_iter

# Open and read the file without guessing
data = json_iter.load_from_file("data.json")

# Process file

# Write to file without guessing
jsonl_iter.dump_to_file(data, "data.jsonl")

import json_streams
def process(data):
    for entry in data:
        # process
        yield entry

def read_process_and_write(filename_in, filename_out):

    json_streams.dump_to_file(
        process(
            json_streams.load_from_file(filename_in)
        ),
        filename_out
    )

Development

After cloning the repo, just run

$ make test

to setup a virtual environment, install dev dependencies and run the unit tests.

Note: If you run the command in a activated virtual environment, that environment is used instead.