ijson 3.5.1

Installation

Ijson is hosted in PyPI, so you should be able to install it via pip:

pip install ijson

Binary wheels are provided for major platforms and python versions. These are built and published automatically using cibuildwheel via GitHub Actions.

Usage

All usage example will be using a JSON document describing geographical objects:

{
  "earth": {
    "europe": [
      {"name": "Paris", "type": "city", "info": { ... }},
      {"name": "Thames", "type": "river", "info": { ... }},
      // ...
    ],
    "america": [
      {"name": "Texas", "type": "state", "info": { ... }},
      // ...
    ]
  }
}

import ijson

f = urlopen('http://.../')
objects = ijson.items(f, 'earth.europe.item')
cities = (o for o in objects if o['type'] == 'city')
for city in cities:
    do_something_with(city)

import ijson

f = urlopen('http://.../')
european_places = ijson.kvitems(f, 'earth.europe.item')
names = (v for k, v in european_places if k == 'name')
for name in names:
    do_something_with(name)

import ijson

parser = ijson.parse(urlopen('http://.../'))
stream.write('<geo>')
for prefix, event, value in parser:
    if (prefix, event) == ('earth', 'map_key'):
        stream.write('<%s>' % value)
        continent = value
    elif prefix.endswith('.name'):
        stream.write('<object name="%s"/>' % value)
    elif (prefix, event) == ('earth.%s' % continent, 'end_map'):
        stream.write('</%s>' % continent)
stream.write('</geo>')

import ijson

events = ijson.basic_parse(urlopen('http://.../'))
num_names = sum(1 for event, value in events
                if event == 'map_key' and value == 'name')

$> echo '{"A": 0, "B": [1, 2, 3, 4]}' | python -m ijson.dump -m parse
#: path, name, value
--------------------
0: , start_map, None
1: , map_key, A
2: A, number, 0
3: , map_key, B
4: B, start_array, None
5: B.item, number, 1
6: B.item, number, 2
7: B.item, number, 3
8: B.item, number, 4
9: B, end_array, None
10: , end_map, None

$> python -m ijson.benchmark my/json/file.json -m items -p values.item

import asyncio
import ijson

async def run():
   f = await async_urlopen('http://..../')
   async for object in ijson.items(f, 'earth.europe.item'):
      if object['type'] == 'city':
         do_something_with(city)
asyncio.run(run())

import io
import ijson

parse_events = ijson.parse(io.BytesIO(b'["skip", {"a": 1}, {"b": 2}, {"c": 3}]'))
while True:
    prefix, event, value = next(parse_events)
    if value == "skip":
        break
for obj in ijson.items(parse_events, 'item'):
    print(obj)

import ijson

@ijson.coroutine
def print_cities():
   while True:
      obj = (yield)
      if obj['type'] != 'city':
         continue
      print(obj)

coro = ijson.items_coro(print_cities(), 'earth.europe.item')
f = urlopen('http://.../')
for chunk in iter(functools.partial(f.read, buf_size)):
   coro.send(chunk)
coro.close()

import ijson

events = ijson.sendable_list()
coro = ijson.items_coro(events, 'earth.europe.item')
f = urlopen('http://.../')
for chunk in iter(functools.partial(f.read, buf_size)):
   coro.send(chunk)
   process_accumulated_events(events)
   del events[:]
coro.close()
process_accumulated_events(events)

Options

Additional options are supported by all ijson functions to give users more fine-grained control over certain operations:

• The use_float option (defaults to False) controls how non-integer values are returned to the user. If set to True users receive float() values; otherwise Decimal values are constructed. Note that building float values is usually faster, but on the other hand there might be loss of precision (which most applications will not care about) and will raise an exception when overflow occurs (e.g., if 1e400 is encountered). This option also has the side-effect that integer numbers bigger than 2^64 (but sometimes 2^32, see capabilities) will also raise an overflow error, due to similar reasons. Future versions of ijson might change the default value of this option to True.

• The multiple_values option (defaults to False) controls whether multiple top-level values are supported. JSON content should contain a single top-level value (see the JSON Grammar). However there are plenty of JSON files out in the wild that contain multiple top-level values, often separated by newlines. By default ijson will fail to process these with a parse error: trailing garbage error unless multiple_values=True is specified.

• Similarly the allow_comments option (defaults to False) controls whether C-style comments (e.g., /* a comment */), which are not supported by the JSON standard, are allowed in the content or not.

• For functions taking a file-like object, an additional buf_size option (defaults to 65536 or 64KB) specifies the amount of bytes the library should attempt to read each time.

• The items and kvitems functions, and all their variants, have an optional map_type argument (defaults to dict) used to construct objects from the JSON stream. This should be a dict-like type supporting item assignment.

Events

When using the lower-level ijson.parse function, three-element tuples are generated containing a prefix, an event name, and a value. Events will be one of the following:

• start_map and end_map indicate the beginning and end of a JSON object, respectively. They carry a None as their value.

• start_array and end_array indicate the beginning and end of a JSON array, respectively. They also carry a None as their value.

• map_key indicates the name of a field in a JSON object. Its associated value is the name itself.

• null, boolean, integer, double, number and string all indicate actual content, which is stored in the associated value.

Prefix

A prefix represents the context within a JSON document where an event originates at. It works as follows:

• It starts as an empty string.

• A <name> part is appended when the parser starts parsing the contents of a JSON object member called name, and removed once the content finishes.

• A literal item part is appended when the parser is parsing elements of a JSON array, and removed when the array ends.

• Parts are separated by ..

When using the ijson.items function, the prefix works as the selection for which objects should be automatically built and returned by ijson.

Backends

Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends:

• yajl2_c: a C extension using YAJL 2.x. This is the fastest, but might require a compiler and the YAJL development files to be present when installing this package. Binary wheel distributions exist for major platforms/architectures to spare users from having to compile the package.

• yajl2_cffi: wrapper around YAJL 2.x using CFFI.

• yajl2: wrapper around YAJL 2.x using ctypes, for when you can’t use CFFI for some reason.

• yajl: deprecated YAJL 1.x + ctypes wrapper, for even older systems.

• python: pure Python parser, good to use with PyPy

This list of backend names is available under the ijson.ALL_BACKENDS constant.

You can import a specific backend and use it in the same way as the top level library:

import ijson.backends.yajl2_cffi as ijson

for item in ijson.items(...):
    # ...

Importing the top level library as import ijson uses the first available backend in the same order of the list above, and its name is recorded under ijson.backend. If the IJSON_BACKEND environment variable is set its value takes precedence and is used to select the default backend.

You can also use the ijson.get_backend function to get a specific backend based on a name:

backend = ijson.get_backend('yajl2_c')
for item in backend.items(...):
    # ...

Performance tips

In more-or-less decreasing order, these are the most common actions you can take to ensure you get most of the performance out of ijson:

• Make sure you use the fastest backend available. See backends for details.

• If you know your JSON data contains only numbers that are “well behaved” consider turning on the use_float option. See options for details.

• Make sure you feed ijson with binary data instead of text data. See faq #1 for details.

• Play with the buf_size option, as depending on your data source and your system a value different from the default might show better performance. See options for details.

The benchmarking tool should help with trying some of these options and observing their effect on your input files.

FAQ

1. Q: Does ijson work with bytes or str values?

   A: In short: both are accepted as input, outputs are only str.

   All ijson functions expecting a file-like object should ideally be given one that is opened in binary mode (i.e., its read function returns bytes objects, not str). However if a text-mode file object is given then the library will automatically encode the strings into UTF-8 bytes. A warning is currently issued (but not visible by default) alerting users about this automatic conversion.

   On the other hand ijson always returns text data (JSON string values, object member names, event names, etc) as str objects. This mimics the behavior of the system json module.

2. Q: How are numbers dealt with?

   A: ijson returns int values for integers and decimal.Decimal values for floating-point numbers. This is mostly because of historical reasons. Since 3.1 a new use_float option (defaults to False) is available to return float values instead. See the options section for details.

3. Q: I’m getting an UnicodeDecodeError, or an IncompleteJSONError with no message

   A: This error is caused by byte sequences that are not valid in UTF-8. In other words, the data given to ijson is not really UTF-8 encoded, or at least not properly.

   Depending on where the data comes from you have different options:

   • If you have control over the source of the data, fix it.

   • If you have a way to intercept the data flow, do so and pass it through a “byte corrector”. For instance, if you have a shell pipeline feeding data through stdin into your process you can add something like ... | iconv -f utf8 -t utf8 -c | ... in between to correct invalid byte sequences.

   • If you are working purely in python, you can create a UTF-8 decoder using codecs’ incrementaldecoder to leniently decode your bytes into strings, and feed those strings (using a file-like class) into ijson (see our string_reader_async internal class for some inspiration).

   In the future ijson might offer something out of the box to deal with invalid UTF-8 byte sequences.

4. Q: I’m getting parse error: trailing garbage or Additional data found errors

   A: This error signals that the input contains more data than the top-level JSON value it’s meant to contain. This is usually caused by JSON data sources containing multiple values, and is usually solved by passing the multiple_values=True to the ijson function in use. See the options section for details.

5. Q: How do I use ijson with requests or httpx

   A: The requests library downloads the body of the HTTP response immediately by default. To stream JSON into ijson, pass stream=True and adapt the byte iterator:

   import requests
   import ijson
   
   with requests.get('https://..', stream=True) as resp:
       resp.raise_for_status()
       f = ijson.from_iter(resp.iter_content(chunk_size=64*1024))
       objects = ijson.items(f, 'earth.europe.item')
       cities = (o for o in objects if o['type'] == 'city')
       for city in cities:
         do_something_with(city)

   You can also pass Response.raw directly (it’s a file-like object), but using iter_content is preferred because requests will transparently handle HTTP transfer encodings (e.g., gzip, chunked).

   For async usage with httpx:

   import httpx
   import ijson
   
   async with httpx.AsyncClient() as client:
       async with client.stream('GET', 'https://..') as resp:
           resp.raise_for_status()
           f = ijson.from_iter(resp.aiter_bytes())
           objects = ijson.items(f, 'earth.europe.item')
           cities = (o async for o in objects if o['type'] == 'city')
           async for city in cities:
             do_something_with(city)

Acknowledgements

ijson was originally developed and actively maintained until 2016 by Ivan Sagalaev. In 2019 he handed over the maintenance of the project and the PyPI ownership.

Python parser in ijson is relatively simple thanks to Douglas Crockford who invented a strict, easy to parse syntax.

The YAJL library by Lloyd Hilaiel is the most popular and efficient way to parse JSON in an iterative fashion. When building the library ourselves, we use our own fork that contains fixes for all known CVEs.

Ijson was inspired by yajl-py wrapper by Hatem Nassrat. Though ijson borrows almost nothing from the actual yajl-py code it was used as an example of integration with yajl using ctypes.