Skip to main content

Stream objects from json-arrays or json lines lazily..

Project description

json-streams

Build Status codecov Build & Publish PyPI status

Read and write JSON lazy, especially json-arrays.

Handles both the JSON format:

[
    {
        "a": 1
    },
    {
        "a": 2
    }
]

As well as JSON LINES format:

{"a":1}
{"a": 2}

Uses orjson or ujson if present, otherwise standard json.

Usage

Installation

# Using standard json
pip install json-streams

# Using orjson
pip install json-streams[orjson]

# Using ujson
pip install json-streams[ujson]

Note

This library prefers files opened in binary mode. Therefore does all dumps-methods return bytes.

All loads methods handles str argument. If you use the orjson library you can also pass bytes or bytesarray to loads. The goal is to have all loads handling str, bytes and bytesarray.

Examples

Allows you to use json.load and json.dump with both json and json-lines files as well as dumping generators.

import json_streams

# This command tries to guess format and opens the file
data = json_streams.load_from_file("data.json") # or data.jsonl

# Write to file, again guessing format
json_streams.dump_to_file(data, "data.jsonl")
from json_streams import json_iter, jsonl_iter

# Open and read the file without guessing
data = json_iter.load_from_file("data.json")

# Process file

# Write to file without guessing
jsonl_iter.dump_to_file(data, "data.jsonl")
import json_streams
def process(data):
    for entry in data:
        # process
        yield entry

def read_process_and_write(filename_in, filename_out):

    json_streams.dump_to_file(
        process(
            json_streams.load_from_file(filename_in)
        ),
        filename_out
    )

Development

After cloning the repo, just run

$ make test

to setup a virtual environment, install dev dependencies and run the unit tests.

Note: If you run the command in a activated virtual environment, that environment is used instead.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for json-streams, version 0.3.1
Filename, size File type Python version Upload date Hashes
Filename, size json_streams-0.3.1-py3-none-any.whl (6.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size json-streams-0.3.1.tar.gz (8.1 MB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page