Incremental JSON Parser

An incremental JSON parser geared towards handling JSON data generated by Large Language Models (LLMs). It is ideal in any situation where you need to process JSON data containing long string values as it arrives in small chunks.

Installation from Git

To install the Incremental JSON Parser, clone the repository and use pip to install the package:

git clone https://github.com/jsh6789/ijp.git
cd ijp
pip install .

Usage

The example below shows how to use the parser in your own scripts:

from ijp import IncrementalJSONParser

json_string = '''
{
    "price": 19.99,
    "itemNo": "3735272",
    "modulars" : [
        {
            "zone": "A",
            "section": 29,
            "position": 10
        },
        {
            "zone": "A",
            "section": 29,
            "position": 15
        }
    ]
}'''

# For demonstratory purpose: break the JSON into chunks.
chunk_size = 4
chunk_list = [
    json_string[i:i + chunk_size] for i in range(0, len(json_string), chunk_size)
]

with IncrementalJSONParser() as parser:
    for chunk in chunk_list:
        parser.send(chunk)
        for token in parser:
            print(token)

This will output:

(['price'], 'float', 19.99)
(['itemNo'], 'stringpart', '37')
(['itemNo'], 'stringpart', '3527')
(['itemNo'], 'stringpart', '2')
(['itemNo'], 'string', '3735272')
(['modulars', 0, 'zone'], 'stringpart', 'A')
(['modulars', 0, 'zone'], 'string', 'A')
(['modulars', 0, 'section'], 'int', 29)
(['modulars', 0, 'position'], 'int', 10)
(['modulars', 1, 'zone'], 'stringpart', 'A')
(['modulars', 1, 'zone'], 'string', 'A')
(['modulars', 1, 'section'], 'int', 29)
(['modulars', 1, 'position'], 'int', 15)

More examples demonstrating uses for this parser can be found in the 'examples' directory.

License

This project is in the public domain. See the accompanying UNLICENSE file for more info.