Skip to main content

Iterative JSON parser with a standard Python iterator interface

Project description

https://travis-ci.com/ICRAR/ijson.svg?branch=master https://coveralls.io/repos/github/ICRAR/ijson/badge.svg?branch=master https://badge.fury.io/py/ijson.svg

ijson

Ijson is an iterative JSON parser with a standard Python iterator interface.

Usage

All usage example will be using a JSON document describing geographical objects:

{
  "earth": {
    "europe": [
      {"name": "Paris", "type": "city", "info": { ... }},
      {"name": "Thames", "type": "river", "info": { ... }},
      // ...
    ],
    "america": [
      {"name": "Texas", "type": "state", "info": { ... }},
      // ...
    ]
  }
}

Most common usage is having ijson yield native Python objects out of a JSON stream located under a prefix. Here’s how to process all European cities:

import ijson

f = urlopen('http://.../')
objects = ijson.items(f, 'earth.europe.item')
cities = (o for o in objects if o['type'] == 'city')
for city in cities:
    do_something_with(city)

For how to build a prefix see the Prefix section below.

Other times it might be useful to iterate over object members rather than objects themselves (e.g., when objects are too big). In that case one can use the kvitems functions instead:

import ijson

f = urlopen('http://.../')
european_places = ijson.kvitems(f, 'earth.europe.item')
names = (v for k, v in european_places if k == 'name')
for name in names:
    do_something_with(name)

Sometimes when dealing with a particularly large JSON payload it may worth to not even construct individual Python objects and react on individual events immediately producing some result:

import ijson

parser = ijson.parse(urlopen('http://.../'))
stream.write('<geo>')
for prefix, event, value in parser:
    if (prefix, event) == ('earth', 'map_key'):
        stream.write('<%s>' % value)
        continent = value
    elif prefix.endswith('.name'):
        stream.write('<object name="%s"/>' % value)
    elif (prefix, event) == ('earth.%s' % continent, 'end_map'):
        stream.write('</%s>' % continent)
stream.write('</geo>')

Events

When using the lower-level ijson.parse function, three-element tuples are generated containing a prefix, an event name, and a value. Events will be one of the following:

  • start_map and end_map indicate the beginning and end of a JSON object, respectively. They carry a None as their value.

  • start_array and end_array indicate the beginning and end of a JSON array, respectively. They also carry a None as their value.

  • map_key indicates the name of a field in a JSON object. Its associated value is the name itself.

  • null, boolean, integer, double, number and string all indicate actual content, which is stored in the associated value.

Prefix

A prefix represents the context within a JSON document where an event originates at. It works as follows:

  • It starts as an empty string.

  • A <name> part is appended when the parser starts parsing the contents of a JSON object member called name, and removed once the content finishes.

  • A literal item part is appended when the parser is parsing elements of a JSON array, and removed when the array ends.

  • Parts are separated by ..

When using the ijson.items function, the prefix works as the selection for which objects should be automatically built and returned by ijson.

Backends

Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends:

  • yajl2_c: a C extension using YAJL 2.x. This is the fastest, but might require a compiler and the YAJL development files to be present when installing this package. Binary wheel distributions exist for major platforms/architectures to spare users from having to compile the package.

  • yajl2_cffi: wrapper around YAJL 2.x using CFFI.

  • yajl2: wrapper around YAJL 2.x using ctypes, for when you can’t use CFFI for some reason.

  • yajl: deprecated YAJL 1.x + ctypes wrapper, for even older systems.

  • python: pure Python parser, good to use with PyPy

You can import a specific backend and use it in the same way as the top level library:

import ijson.backends.yajl2_cffi as ijson

for item in ijson.items(...):
    # ...

Importing the top level library as import ijson uses the first available backend in the same order of the list above.

Acknowledgements

ijson was originally developed and actively maintained until 2016 by Ivan Sagalaev. In 2019 he handed over the maintenance of the project and the PyPI ownership.

Python parser in ijson is relatively simple thanks to Douglas Crockford who invented a strict, easy to parse syntax.

The YAJL library by Lloyd Hilaiel is the most popular and efficient way to parse JSON in an iterative fashion.

Ijson was inspired by yajl-py wrapper by Hatem Nassrat. Though ijson borrows almost nothing from the actual yajl-py code it was used as an example of integration with yajl using ctypes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ijson-2.6.1.tar.gz (29.4 kB view details)

Uploaded Source

Built Distributions

ijson-2.6.1-cp37-cp37m-manylinux1_x86_64.whl (65.7 kB view details)

Uploaded CPython 3.7m

ijson-2.6.1-cp37-cp37m-macosx_10_6_x86_64.whl (25.5 kB view details)

Uploaded CPython 3.7mmacOS 10.6+ x86-64

ijson-2.6.1-cp36-cp36m-manylinux1_x86_64.whl (65.7 kB view details)

Uploaded CPython 3.6m

ijson-2.6.1-cp36-cp36m-macosx_10_6_x86_64.whl (25.5 kB view details)

Uploaded CPython 3.6mmacOS 10.6+ x86-64

ijson-2.6.1-cp35-cp35m-manylinux1_x86_64.whl (65.8 kB view details)

Uploaded CPython 3.5m

ijson-2.6.1-cp35-cp35m-macosx_10_6_x86_64.whl (25.5 kB view details)

Uploaded CPython 3.5mmacOS 10.6+ x86-64

ijson-2.6.1-cp34-cp34m-manylinux1_x86_64.whl (65.6 kB view details)

Uploaded CPython 3.4m

ijson-2.6.1-cp34-cp34m-macosx_10_6_x86_64.whl (25.5 kB view details)

Uploaded CPython 3.4mmacOS 10.6+ x86-64

ijson-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl (63.5 kB view details)

Uploaded CPython 2.7mu

ijson-2.6.1-cp27-cp27m-manylinux1_x86_64.whl (63.5 kB view details)

Uploaded CPython 2.7m

ijson-2.6.1-cp27-cp27m-macosx_10_6_x86_64.whl (25.4 kB view details)

Uploaded CPython 2.7mmacOS 10.6+ x86-64

File details

Details for the file ijson-2.6.1.tar.gz.

File metadata

  • Download URL: ijson-2.6.1.tar.gz
  • Upload date:
  • Size: 29.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1.tar.gz
Algorithm Hash digest
SHA256 75ebc60b23abfb1c97f475ab5d07a5ed725ad4bd1f58513d8b258c21f02703d0
MD5 bf166826194c81b351361d141e203c7e
BLAKE2b-256 d0123116e1d5752aa9d480eb58ae4b348d38c1aeaf792c5fbca22e44c27d4bf1

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 65.7 kB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 25d4d159405f75a7443c1fe83b6d7be5a7da017b4aa9cc1bb5cda3feb74aaf32
MD5 73ce0310a9df14fa524ffa552b03cbef
BLAKE2b-256 191d7c97e1dd574667bbaefc9540feafbcdb8c0641ec2c88410c174b7b95e2cb

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp37-cp37m-macosx_10_6_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp37-cp37m-macosx_10_6_x86_64.whl
  • Upload date:
  • Size: 25.5 kB
  • Tags: CPython 3.7m, macOS 10.6+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp37-cp37m-macosx_10_6_x86_64.whl
Algorithm Hash digest
SHA256 26978c02314233c87bddad8800b7b9a56a052334f495e2bce93b282397c6931d
MD5 060161d87dd0956b59489ee44a18507e
BLAKE2b-256 fcd88966b7f32166e04a336a35a49867b39e45f92e3d34849b5f4a3aa3a3b4ce

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 65.7 kB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 c0042bb768fb890c177af923c0ead157cdc70c6dfa64827765c1a3676a879190
MD5 de9576b1dfa093379438aead61dd21d0
BLAKE2b-256 23422066f77a714ab7221542ea23710b35a96c5dd398f1933429088afd888293

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp36-cp36m-macosx_10_6_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp36-cp36m-macosx_10_6_x86_64.whl
  • Upload date:
  • Size: 25.5 kB
  • Tags: CPython 3.6m, macOS 10.6+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp36-cp36m-macosx_10_6_x86_64.whl
Algorithm Hash digest
SHA256 a8b486bdf24e389947e588f4021498f6cc56cafdfaec1c78e9952e0f338aef23
MD5 30e592e378f7912bbe01f96c7e90074a
BLAKE2b-256 3178f13a7c5a7b14974348bdcd7879890bcd847e6cfb8252087297f0a9d31eba

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 65.8 kB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8ce67b7d3435c3fc831d5c06f60b2d20a853b599cdf885478e575a3416fbf655
MD5 c6750865e364b64fd7742c97d8b33cd8
BLAKE2b-256 20064c28511c77b658d58917b7057e42d447353e9d29cd6d8ff8f0f00c37ac2f

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp35-cp35m-macosx_10_6_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp35-cp35m-macosx_10_6_x86_64.whl
  • Upload date:
  • Size: 25.5 kB
  • Tags: CPython 3.5m, macOS 10.6+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp35-cp35m-macosx_10_6_x86_64.whl
Algorithm Hash digest
SHA256 ae9cc3ebbe8fa030b923b5dff912a61980edd03dc00b92f5c0223e44cbc51d9f
MD5 11ef6e745a8850d8b3b7ec74fd12247b
BLAKE2b-256 2834ea5383be46d1e6231d4958d3b84a3c94935b256ffaa7cb01da132a055448

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 65.6 kB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 7bac04b23691e6ab122d8f9ff06b26dbbb6df01babbf6bf8856ccad1c505278b
MD5 3e897b681aee834cab40ec6454858a51
BLAKE2b-256 f6d3494a24842f4595766faf07b4530539b01d9fc12a2e4ba9e8d048e5ba59e8

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp34-cp34m-macosx_10_6_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp34-cp34m-macosx_10_6_x86_64.whl
  • Upload date:
  • Size: 25.5 kB
  • Tags: CPython 3.4m, macOS 10.6+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp34-cp34m-macosx_10_6_x86_64.whl
Algorithm Hash digest
SHA256 9904bf55bc1f170353c32144861d8295f0bdc41034e5e6ae58cbf30610023ca6
MD5 4b937637825294436a6bc83669baf17b
BLAKE2b-256 7beed07ebb0f02a57162979ce40f207680cf68d55d11b3a75df8116f1197b8a8

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 63.5 kB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a4cd7f8ecf035d0e23db1cc6d6036e6c563f31abacbceae88904bb8b7f88b1f6
MD5 7ec56b1d4f1f6df16b32a98b6c669d5c
BLAKE2b-256 53fc207cd7a039975e6073185fe2012f2ecdfaee5b65727edf92bd801235630c

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 63.5 kB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d320dc1c1c9adbe404668b0fed6bfa0ac8693911159564f4655a5f2059746993
MD5 c291eabc583546797af6145904f04fa0
BLAKE2b-256 1d7c1ea72b0777b70246a54e6e7904c17cb1b2d73bf00b72b4c06cd55648eda5

See more details on using hashes here.

File details

Details for the file ijson-2.6.1-cp27-cp27m-macosx_10_6_x86_64.whl.

File metadata

  • Download URL: ijson-2.6.1-cp27-cp27m-macosx_10_6_x86_64.whl
  • Upload date:
  • Size: 25.4 kB
  • Tags: CPython 2.7m, macOS 10.6+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.42.0 CPython/3.8.0

File hashes

Hashes for ijson-2.6.1-cp27-cp27m-macosx_10_6_x86_64.whl
Algorithm Hash digest
SHA256 60393946d73792d5adeeaa25e82ff2f5bf19b17f6617a468743a4db4a07298a0
MD5 7fece24e9ea5e8cf8b9ead1af2673666
BLAKE2b-256 f3f63110ff5af7e0658720aabf8b9f24c8beb7e71b5f4b89143cd0e43c73eed5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page