shaped-nujson 1.35.2

Ultra fast JSON encoder and decoder for Python with NumPy support

nujson: The UltraJSON Fork That Supports Numpy Serialization That Actually Includes Python Wheels

Inspired by Pandas' ujson

• How to install
• Example
• Why make such a package and what is modified

How to install

Python versions: Python 3.5+

Using pip:

pip install shaped-nujson

If you get an error like this:

ERROR: Could not find a version that satisfies the requirement numpy>=1.16.4 (from nujson) (from versions: 1.9.3)
ERROR: No matching distribution found for numpy>=1.16.4 (from nujson)

Try this:

pip uninstall numpy
pip install numpy==1.16.4
pip install shaped-nujson

Example

>>> import numpy as np
>>> import nujson as ujson
>>> a = {"a": np.int64(100)}
>>> ujson.dumps(a)
'{"a":100}'
>>> a["b"] = np.float64(10.9)
>>> ujson.dumps(a)
'{"a":100,"b":10.9}'
>>> a["c"] = np.str_("12")
>>> ujson.dumps(a)
'{"a":100,"b":10.9,"c":"12"}'
>>> a["d"] = np.array(list(range(10)))
>>> ujson.dumps(a)
'{"a":100,"b":10.9,"c":"12","d":[0,1,2,3,4,5,6,7,8,9]}'
>>> a["e"] = np.repeat(3.9, 4)
>>> ujson.dumps(a)
'{"a":100,"b":10.9,"c":"12","d":[0,1,2,3,4,5,6,7,8,9],"e":[3.9,3.9,3.9,3.9]}'

Why make such a package and what is modified?

On Python 3, some data types of NumPy are not serializable. Here are some references:

• python - Why are some numpy datatypes JSON serializable and others not? - Stack Overflow
• Maximum recursion level reached in Python 3 - Issue #221 - esnme/ultrajson
• Issue 24313: json fails to serialise numpy.int64 - Python tracker

One solution is type conversion like: int(numpy.int64) and numpy.array.tolist(). But it's not good for performance. After searching the Internet, we found an unmaintained project Komnomnomnom/ultrajson which recommends to use Pandas' ujson.

We tried but found Pandas is too heavy for our projects. So we decided to build our own lightweight fork. Currently, esn's ujson master branch has some problems we needed to solve, and the master branch is based on the the v1.35 ujson.

The main point is to convert NumPy data types in C, with calling NumPy's header. Commit 187bd15 has the most changes we made to support NumPy, and commit afedc42 fixes a build issue on macOS caused by Clang.