fastsimdjson

Fast JSON for Python, built on simdjson, the parser that reads JSON at gigabytes per second. fastsimdjson is a drop-in replacement for the json module (loads, load, dumps, dump) that gives the same results 3.5 to 4.2 times faster, and it goes further: lazy parsing, streams of documents on many threads, Apache Arrow tables and MessagePack, in one package with no dependencies.

How it compares

fastsimdjson is often two to four times faster than the alternatives:

Against the standard json module, loads is 3.5 times faster and parse 25 times faster. The files, the versions and the scripts are under Benchmarks.

• Same results. loads returns what json.loads returns, dumps the same str as json.dumps, character for character, and dumpb the same bytes as orjson.dumps. Switching changes the speed, not the output.
• Typed. loads(data, type=T) builds dataclasses, TypedDicts and other typed values straight from the parser, with the types checked: 1.5 to 2.1 times as fast as msgspec, 3.9 to 5.1 times as fast as pydantic.
• Only what you need. parse gives lazy views and builds only the values you read; fields= extracts only some fields from each document of a stream.
• Many threads. On free-threaded Python, streams of documents (NDJSON, JSON Lines) are read and written on many threads: up to 20 GB/s on 64 cores. Arrow and MessagePack conversions release the GIL and use many threads on any Python: on 32 cores, loads_arrow reads NDJSON into Arrow at up to 27.5 GB/s, 2.3 to 2.9 times as fast as pyarrow.
• Arrow without pyarrow. loads_arrow and dumpb_arrow convert NDJSON to Arrow tables and back, for pyarrow, polars or DuckDB, without a copy.
• MessagePack without Python objects. json_to_msgpack and msgpack_to_json convert directly, without building Python objects.
• Minify without parsing. minify strips insignificant whitespace and returns bytes, without building Python objects.
• Everywhere. No dependencies; wheels for Linux (x64 and ARM), macOS (Intel and Apple silicon) and Windows, Python 3.10 to 3.15, including free-threaded Python.

Installation

With pip:

pip install fastsimdjson

With uv:

uv pip install fastsimdjson      # in a uv project: uv add fastsimdjson

Wheels are available for Linux, macOS and Windows, for Python 3.10 to 3.15, including free-threaded Python 3.14 and 3.15.

Wheels are also available for Pyodide (WebAssembly, in the browser or in Node), for Python 3.13 to 3.15:

import micropip
await micropip.install("fastsimdjson")

Under Pyodide, Python cannot start threads: threads= runs on the calling thread.

Usage

loads: the whole document

import fastsimdjson
fastsimdjson.loads(b'{"a": [1, 2.5, "x", true, null]}')
# {'a': [1, 2.5, 'x', True, None]}

loads(data) accepts bytes, bytearray, memoryview and str, and returns the same value as json.loads, with the same types and key order. Integers that do not fit in 64 bits become exact Python ints. NaN, Infinity and -Infinity are accepted, in any capitalization.

Invalid input raises fastsimdjson.JSONDecodeError, a subclass of json.JSONDecodeError. When simdjson rejects a document, the same input is parsed with json.loads. If that succeeds, loads returns its value (this is how a number that overflows a double becomes inf). If it raises JSONDecodeError, the exception is re-raised with Python's message and byte position. Any other exception from json.loads propagates.

parse: lazy views

When you need only part of a document, parse avoids building the rest. It accepts the same inputs as loads and returns read-only views: fastsimdjson.Object (a Mapping) and fastsimdjson.Array (a Sequence). Values are converted when you access them; nested objects and arrays are returned as views. A scalar root is returned as a plain value.

doc = fastsimdjson.parse(open("twitter.json", "rb").read())
ids = [(s["id"], s["user"]["screen_name"]) for s in doc["statuses"]]
doc.at_pointer("/statuses/0/user/name")   # JSON Pointer (RFC 6901)
doc["search_metadata"].as_dict()          # convert a subtree, like loads

Object supports obj[key], get, in, len, iteration over the keys, keys(), values(), items() (iterators), at_pointer and as_dict(). Array supports arr[i] (negative indexes and slices), len, iteration, at_pointer and as_list(). Both work with match statements.

• A view keeps its document alive; the document owns its own buffers, so it remains valid while other documents are parsed.
• A key lookup scans the object. With duplicate keys, lookups return the first value, whereas as_dict() (like json.loads) keeps the last.
• Indexing an array walks it from the last index reached, so a loop over arr[i] is linear; iteration is the fastest way to visit an array.
• Errors are handled as in loads. A document that simdjson rejects but json.loads accepts (an overflowing number, an unpaired surrogate) is returned as plain Python objects, as loads would return it.

type=: typed decoding

loads(data, type=T) returns the document as an instance of T, checked on the way. It builds dataclasses straight from the simdjson tape, without an intermediate dict, and is 1.5 to 2.1 times as fast as msgspec on one core (see Typed decoding).

from dataclasses import dataclass, field
from typing import Literal, Optional

@dataclass
class User:
    id: int
    login: str
    followers: int = 0

@dataclass
class Repo:
    id: int
    name: str
    owner: User
    topics: list[str] = field(default_factory=list)
    visibility: Literal["public", "private"] = "public"
    license: Optional[str] = None

repo = fastsimdjson.loads(data, type=Repo)
repos = fastsimdjson.loads(data, type=list[Repo])
events = list(fastsimdjson.loads_many(ndjson, type=Repo))

type accepts:

• int, float (an integer becomes a float), str, bool, type(None) (JSON null; None too inside a type, as in list[None]), and Any or object (no check: the value as loads returns it). type=None, the default, means no type;
• list, tuple (fixed length or tuple[T, ...]), set, frozenset, dict with str keys, and their abstract counterparts (Sequence, Mapping, ...);
• Optional[T], unions (int | str, at most one type per JSON kind; an integer goes to int rather than to float) and Literal (of str, int, bool and None: JSON 1 is not True);
• enums (by value: str, int, bool or None), dataclasses (including frozen ones and those with __slots__), TypedDict (from typing or typing_extensions) and NamedTuple (from an array);
• Annotated, NewType and type aliases, and recursive types.

Dataclass fields take their defaults (and default factories) when the key is missing, keys that are not fields are ignored, __post_init__ runs, and init=False fields get their default. The fields are set as the generated __init__ sets them: an __init__ of your own is not called. A TypedDict becomes a dict with its keys only. A value of the wrong type raises fastsimdjson.ValidationError (a ValueError) with its path:

ValidationError: Expected `int`, got `str` - at `$.owner.id`

A ValueError or TypeError raised by __post_init__ becomes a ValidationError too, with the path. Invalid JSON raises JSONDecodeError, and the documents that only json accepts (NaN, a number that overflows a double) are converted the same way, from what json returns; past what Python's stack allows, those raise RecursionError, as json does. A set of values that cannot be hashed (set[list[int]]) or a union that contains itself is a TypeError. The description of each type is computed once and cached (the cache starts over past 1024 types, so that types made on the fly do not pile up). load, load_file, loads_many and map_many accept type= too; a ValidationError ends the iteration of loads_many, as any error does. With threads=N on free-threaded Python, the instances are built on N threads (not with fields= or lazy=True).

dumps and dump: like json.dumps

fastsimdjson.dumps({"a": [1, 2.5, None]})            # '{"a": [1, 2.5, null]}'
fastsimdjson.dumps(obj, indent=2, sort_keys=True)
with open("out.json", "w", encoding="utf-8") as f:
    fastsimdjson.dump(obj, f)

dumps(obj, **kw) takes the arguments of json.dumps and returns the same str, character for character, including the float format (repr), the escapes and the default separators. ensure_ascii, indent, separators, sort_keys, allow_nan and default are handled in C. Everything else is passed to json.dumps itself, which produces the result or raises its usual exception: a cls argument or other encoder options, skipkeys, a circular reference, NaN with allow_nan=False, a key or a value that json cannot serialize. dump(obj, fp, **kw) writes dumps(obj, **kw) to fp.

dumpb: like orjson.dumps

fastsimdjson.dumpb({"a": [1, 2.5, None]})            # b'{"a":[1,2.5,null]}'
fastsimdjson.dumpb(obj, option=fastsimdjson.OPT_INDENT_2 | fastsimdjson.OPT_SORT_KEYS)
fastsimdjson.dumpb({1, 2}, default=sorted)           # b'[1,2]'

dumpb(obj, default=None, option=None) has the arguments, the output and the errors of orjson.dumps (orjson 3.12): orjson.dumps(obj, ...) can be replaced by fastsimdjson.dumpb(obj, ...) for the types below. It returns compact UTF-8 bytes, writes floats as orjson does (1e-6, 1e+16, NaN and infinities as null), and raises TypeError with orjson's messages: integers beyond 64 bits, a dict key that is not a str, invalid UTF-8 (a lone surrogate), nesting deeper than 254, a type it cannot serialize. orjson.JSONEncodeError is a subclass of TypeError, so except TypeError catches the errors of both.

It serializes str, int, float, bool, None, dict, list, tuple, enums, and subclasses of str, int, dict and list. Anything else goes to default, as in orjson: its result is serialized in place of the object, and an exception it raises becomes the __cause__ of the TypeError. The options are exported under orjson's names and values (OPT_APPEND_NEWLINE, OPT_INDENT_2, OPT_NON_STR_KEYS, OPT_PASSTHROUGH_SUBCLASS, OPT_SORT_KEYS, OPT_STRICT_INTEGER, ...). Unlike orjson, dumpb does not serialize dataclasses, datetime, date, time, UUID, numpy arrays or orjson.Fragment itself: they go to default, as if OPT_PASSTHROUGH_DATACLASS and OPT_PASSTHROUGH_DATETIME were set, and the options that concern only these types are accepted and have no effect.

minify: strip whitespace without parsing

fastsimdjson.minify(b'{ "a" : [1, 2] }')    # b'{"a":[1,2]}'

minify(data) removes insignificant whitespace (spaces, tabs, newlines and carriage returns outside of strings) and returns bytes. It accepts bytes, bytearray, memoryview and str. It does not parse the document and does not validate it: numbers, escapes, duplicate keys and key order are left as they are, and input that is not valid JSON is minified the same way. An unterminated string raises JSONDecodeError (the minifier does not report where the string started, so the position is 0). The call releases the GIL.

dumpb_many: NDJSON, on several threads

data = fastsimdjson.dumpb_many(records)               # b'{...}\n{...}\n'
data = fastsimdjson.dumpb_many(records, threads=16)   # free-threaded Python

dumpb_many(records, default=None, option=None, *, threads=None) returns the records (a list, a tuple or any iterable) as NDJSON bytes: each record as dumpb writes it, followed by a newline, so that the output reads back with loads_many(data, format="lines"). default and option are those of dumpb, except OPT_INDENT_2, which would spread a record over several lines (ValueError); OPT_APPEND_NEWLINE changes nothing.

On free-threaded Python, threads=N writes chunks of records on N threads at once (threads=0 or None: one per processor), with the worker threads of map_many. The output and the errors are those of one thread: when records cannot be serialized, the error of the first of them is raised (default may then have been called on later records, in other threads, and it must be safe to call from several threads at once). With the GIL, the calling thread does all the work. Like dumpb, dumpb_many does not crash when other threads change the records meanwhile, but the output then reflects some of the changes and not others.

Files

with open("data.json", "rb") as f:
    doc = fastsimdjson.load(f)          # like json.load: f.read(), then loads
doc = fastsimdjson.load_file("data.json")
view = fastsimdjson.parse_file("data.json")   # lazy, like parse

load_file(path) and parse_file(path) accept a str, bytes or os.PathLike path and raise OSError (e.g. FileNotFoundError) when the file cannot be read.

loads_many and parse_many: streams of documents

for record in fastsimdjson.loads_many(open("log.ndjson", "rb").read()):
    ...
for view in fastsimdjson.parse_many(data):    # lazy views, like parse
    ...

Both return an iterator over the documents of data (bytes, bytearray, memoryview or str). The format keyword selects how documents are separated:

format	input
"whitespace" (default)	documents separated by white space, including NDJSON and JSON Lines
"lines"	one document per line (NDJSON, JSON Lines)
"json_seq"	RFC 7464 JSON text sequences (each document preceded by \x1e)
"comma"	documents separated by commas: {...}, {...} (simdjson also accepts white space between them)
"array"	the elements of one array: [{...}, {...}]

To read a file without copying it, map it in memory:

import mmap
with open("log.ndjson", "rb") as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
    names = fastsimdjson.map_many(fn, memoryview(m), threads=64)

A memoryview of bytes, or of a file mapped with mmap.ACCESS_READ (as here), is read in place, and the mapped file must not change meanwhile; other buffers (a bytearray, a view of a writable mmap) are copied first.

simdjson parses the input in batches (batch_size, 1 MB by default); a larger document is handled automatically. In every format, documents that simdjson rejects are handled as in loads: a document that json accepts is returned, otherwise JSONDecodeError reports json's message and the position in the whole input. A truncated last document is an error. The views returned by parse_many remain valid after the iterator moves on.

Streams on several threads: threads=N and map_many

On free-threaded Python (3.14t and newer), a stream can be parsed by several threads at once:

docs = fastsimdjson.loads_many(data, threads=16)        # a list, in order
views = fastsimdjson.parse_many(data, threads=16)       # a list of views
# fn runs in the threads too, on each document; its results, in order:
scores = fastsimdjson.map_many(score, data, threads=16)

With threads, loads_many and parse_many return a list of all the documents instead of an iterator. map_many(fn, data, **kw) returns [fn(doc) for doc in loads_many(data, **kw)] (parse_many with lazy=True), with fn called in the threads. threads=0 (or None) uses one thread per processor; small inputs use fewer threads (about one per 512 KB). The workers share one counter. Each claims the next 256 KB and moves both ends forward to a newline (to the next \x1e for json_seq); the pieces are adjacent. On lines and json_seq a document is never cut, and one longer than 256 KB is parsed with the piece that started before it. threads works with the formats "whitespace", "lines" and "json_seq".

The results, and the exceptions, are those of one thread: when a chunk contains an error or a document that only json accepts, the documents before it are kept and the rest is parsed in the calling thread, as without threads. An exception raised by fn is raised again, for the first document where it happened; fn may then have been called on later documents, in other threads, and it must be safe to call from several threads at once.

The worker threads are started by the first call that needs them and kept for later calls (they wait, detached from the interpreter, between calls, and stop at exit). map_many scales best: each document is built, used and freed in the same thread, and only fn's results come back. loads_many must return every document, and the Python objects of all of them, to the calling thread. With the GIL (the default build of Python), the calling thread does all the work: threads has no effect on speed.

With drop_none=True, map_many leaves out the results that are None: when fn keeps only a few documents, the threads do not build a list entry for each of the others, and the calling thread has fewer to collect.

def starred(event):
    kind, repo = event
    return repo if kind == "WatchEvent" else None

names = fastsimdjson.map_many(starred, data, threads=64, drop_none=True,
                              fields=["/type", "/repo/name"])

Only some fields: fields=

When you need a few values from each document, fields extracts them without building the documents: only the values found become Python objects.

events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
# (type, action) for each document

def count(t):
    kind, action, language = t
    ...
fastsimdjson.map_many(count, data, threads=64,
                      fields=["/type", "/payload/action",
                              "/payload/pull_request/base/repo/language"])

loads_many and map_many accept fields: a JSON Pointer (RFC 6901), or a list or tuple of them. Each document then becomes the value at the pointer, or a tuple of the values at the pointers, in order. A value that is an object or an array is built in full. A pointer that leads nowhere (a missing key, an index past the end, or a step into a scalar) gives default, None unless you pass another value; a JSON null gives None. As with at_pointer, a key that appears twice in an object refers to its first occurrence. fields cannot be combined with lazy=True.

With the formats "whitespace" and "lines", fields reads the documents with simdjson's On-Demand API: the top level of each document, the keys on the way to the pointers and the values found there are read and validated, so a malformed or incomplete document is still an error. The values that no pointer reaches are skipped without being checked, so an invalid value elsewhere in a document ({"a": 1, "b": tru} with fields="/a") goes unnoticed. A value that is read is handled as without fields: an error raises JSONDecodeError, and a document that only json accepts is decoded by json. The whole documents are parsed (and validated) with the other formats, and when a pointer is "" or leads into the value of another pointer (["/a", "/a/b"]).

On 7.6 GB of GitHub events, map_many with three fields ran at 2.2 GB/s on one core of an Intel Xeon Gold 6548N, against 1.8 GB/s with lazy views (lazy=True). With 64 threads (free-threaded Python 3.14.10t), it ran at 84 GB/s (lazy views: 76 GB/s).

Arrow: loads_arrow, load_arrow_file and dumpb_arrow

import pyarrow as pa
table = pa.table(fastsimdjson.loads_arrow(data))             # inferring the schema
schema = pa.schema([("id", pa.int64()), ("name", pa.string())])
table = pa.table(fastsimdjson.load_arrow_file("log.ndjson", schema, threads=8))
frame = polars.DataFrame(fastsimdjson.loads_arrow(data, schema))
data = fastsimdjson.dumpb_arrow(table)                       # b'{"id":1,"name":"x"}\n...'
data = fastsimdjson.dumpb_arrow(frame, threads=8)            # a polars.DataFrame

loads_arrow(data, schema=None, *, block_size=1048576, error_on_unknown_fields=False, validate_skipped=True, threads=None) reads NDJSON (one object per record) into Arrow record batches. The input is bytes, bytearray, memoryview or str, read in place. load_arrow_file(path, ...) maps the file in memory and reads it the same way (the file must not shrink meanwhile). Both return a fastsimdjson.ArrowStream, which goes to any library that supports the Arrow PyCapsule interface, without a copy: pyarrow.table(s), pyarrow.RecordBatchReader.from_stream(s), polars.DataFrame(s), duckdb.from_arrow(s)... Like a RecordBatchReader, it hands over its batches once (ValueError afterwards); pyarrow.schema(s) gives its schema at any time, and s.num_rows the number of records (None once the batches are taken). fastsimdjson does not depend on pyarrow.

• The schema is a pyarrow.Schema (or any object with __arrow_c_schema__) whose fields are booleans, 64-bit integers, unsigned 64-bit integers, doubles, strings, timestamps, structs and lists of these, or null. A field absent from a record is null; a key absent from the schema is skipped, or an error with error_on_unknown_fields=True. Skipped values are checked to be valid JSON, unless validate_skipped=False (then only their structure is).
• Without a schema, it is inferred as pyarrow's JSON reader does: null, true/false, integers that fit in 64 bits (int64), other numbers (double), strings, objects (structs, with the union of their fields in order of first appearance) and arrays (lists). A field of nulls takes the type of its other values and integers become doubles next to other numbers; other mixes of types raise ValueError. Strings stay strings (no timestamps).
• A timestamp field (pa.timestamp("us", tz="UTC"), any unit, with or without a time zone) reads ISO 8601 strings, such as the created_at of GH Archive events ("2015-01-01T15:00:00Z"): a date (a four-digit year), then optionally a time (T or a space, then 15, 15:00, 15:00:00 or 15:00:00.123456), then optionally Z or an offset (+05:00). A time without Z or an offset is UTC. An integer is a count of units since 1970. As with pyarrow.json.read_json (which returns the same table from the strings it accepts; a lowercase t or z and a comma before the fraction are accepted here too), a fraction with more digits than the unit raises ValueError, as does an invalid date. Reading the timestamps costs next to nothing: GH Archive with created_at as timestamp[us, UTC] reads at 2.11 GB/s on one core, as with strings, against 0.45 GB/s for pyarrow and 0.21 GB/s for polars.
• A new batch starts every block_size bytes of input. With threads=N (threads=0 or None: one per processor), N threads read blocks of block_size bytes, cut at line ends: a record must then fit on one line. When that fails, the input is read again on one thread, so that the errors (ValueError, with the record number and the field) are the same with any number of threads.

dumpb_arrow(data, *, write_nulls=True, threads=None) writes Arrow data as NDJSON bytes, one object per row: a pyarrow.Table, RecordBatch, RecordBatchReader or StructArray, a polars.DataFrame, a DuckDB relation, an ArrowStream, or any object with __arrow_c_stream__ or __arrow_c_array__. It writes integers of 8 to 64 bits, floats and doubles (as dumpb writes them, NaN and infinities as null), booleans, strings (including large strings and string views), timestamps (ISO 8601 in UTC, "2015-01-01T15:00:00.250000Z", with Z when the timestamp has a time zone and the fraction when it is not zero), dates ("2015-01-01"), structs and lists (including large and fixed-size lists); other types raise ValueError. With write_nulls=False, null fields are omitted. With threads=N, N threads write ranges of rows into the result. The output reads back with loads_arrow, and with loads_many(data, format="lines"), except for timestamps and dates outside the years 0000 to 9999: their year is written with more digits or a minus sign, which loads_arrow does not read (nor does pyarrow).

These functions release the GIL and run their threads outside Python, with or without free threading. They are built on simdjson_arrow, a C++ library (vendored).

MessagePack: json_to_msgpack and msgpack_to_json

packed = fastsimdjson.json_to_msgpack(b'{"id": 1, "tags": ["a", "b"], "score": 2.5}')
# b'\x83\xa2id\x01\xa4tags\x92\xa1a\xa1b\xa5score\xcb@\x04\x00\x00\x00\x00\x00\x00'
msgpack.unpackb(packed)              # {'id': 1, 'tags': ['a', 'b'], 'score': 2.5}
fastsimdjson.msgpack_to_json(packed)  # b'{"id":1,"tags":["a","b"],"score":2.5}'
stream = fastsimdjson.json_to_msgpack(ndjson, ndjson=True, threads=8)  # one value per document
lines = fastsimdjson.msgpack_to_json(stream, stream=True, threads=8)   # NDJSON again

json_to_msgpack(data, *, ndjson=False, block_size=1048576, threads=None) converts JSON (bytes, bytearray, memoryview or str, read in place) to MessagePack bytes in the most compact encoding: the same bytes as msgpack.packb(json.loads(data)) (but for duplicate keys, which json.loads drops, and integers beyond 64 bits). Integers take the smallest integer encoding (integers beyond 64 bits become doubles), other numbers are doubles, and strings, arrays and objects take the smallest header for their size; keys keep their order, duplicates too. With ndjson=True, the input holds documents separated by whitespace (NDJSON, JSON Lines), and each becomes one MessagePack value, one after the other.

msgpack_to_json(data, *, stream=False, validate_utf8=True, binary_as_base64=True, ext_as_null=False, block_size=1048576, threads=None) converts MessagePack (any bytes-like object) to compact JSON bytes. Floats get the shortest representation that gives them back, as Python's repr (1.0, not 1; 1e-05; NaN and infinities as null), binary values as base64 strings, map keys that are not strings as strings ("1", "true", "null"); extension values (timestamps too) raise ValueError, or become null with ext_as_null=True. Strings must be valid UTF-8 (unless validate_utf8=False). With stream=True, the input holds a sequence of values (what json_to_msgpack(..., ndjson=True) or several msgpack.packb calls give), and each becomes one line of NDJSON.

• Invalid input raises ValueError: "TAPE_ERROR: ... (document 12)" for JSON, "truncated string (byte 1234)" for MessagePack.
• With threads=N (threads=0 or None: one per processor), NDJSON and streams are converted on N threads, in blocks of about block_size bytes. A JSON document must then fit on one line; when that fails, the input is converted again on one thread, so that errors are the same with any number of threads. Single documents use one thread.
• The functions release the GIL and run their threads outside Python, with or without free threading, and write their result straight into the bytes object (no copy). No Python object is made from the data.

They are built on simdjson_msgpack, a C++ library (vendored).

Threads and free-threaded Python

The module is marked free-threading compatible (Py_MOD_GIL_NOT_USED on Python 3.13 and newer), so importing it on a free-threaded build does not re-enable the GIL. It is tested with many threads parsing, serializing and sharing views and stream iterators on free-threaded Python 3.14 and 3.15.

bytes and str inputs are read in place. A bytearray or memoryview could be changed by another thread while we read it, so it is copied first: always by loads_many, parse_many and map_many (the iterator outlives the call) and by loads_arrow, json_to_msgpack and msgpack_to_json (which release the GIL), and only on a free-threaded build by loads and parse. The exception: loads_many, parse_many and map_many read a read-only, contiguous memoryview of bytes, or of a file mapped with mmap.ACCESS_READ, in place (the file must not change meanwhile). Other read-only views (memoryview(bytearray).toreadonly(), a read-only view of a writable mmap) are copied, since what they show can still change. Likewise, on a free-threaded build, dumps and dumpb hold references to the items of the containers they serialize.

release

release() frees the simdjson parser and the string caches kept by the calling thread. On free-threaded Python, it also has the idle worker threads of the parallel streams (threads=N) free theirs; a worker busy with a stream of another thread does so when it is done. Views returned by parse remain valid.

Build and test

Python 3.10 or newer, and a C++17 compiler (clang, GCC or MSVC). The simdjson 5.1.1 and simdutf 9.2.1 amalgamations, simdjson_arrow 0.1.5 (NDJSON to Arrow and back: simdjson_arrow.h and simdjson_arrow.cpp) and simdjson_msgpack 0.1.1 (JSON to MessagePack and back: simdjson_msgpack.h and simdjson_msgpack.cpp), both under the Apache-2.0 license, are already in vendor/. simdjson_arrow includes zmij 1.2 (shortest float formatting, MIT license), which simdjson_msgpack, dumps and dumpb also use (vendor/zmij.h). The tests of the Arrow functions (tests/test_arrow.py) run when pyarrow is installed (and polars, for one of them); otherwise they are skipped.

pip:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip setuptools
python -m pip install -e ".[test]"
pytest tests

uv:

uv venv
. .venv/bin/activate
uv pip install -e ".[test]"
pytest tests

Either one builds the extension and makes import fastsimdjson work in the virtualenv. uv uses the setuptools build requirement from pyproject.toml, so it does not need a separate setuptools install for this path.

To compile the extension in the tree instead, install setuptools and pytest into the same virtualenv, then:

python -m pip install setuptools pytest
python setup.py build_ext --inplace
PYTHONPATH=src python -m pytest tests

uv pip install setuptools pytest
python setup.py build_ext --inplace
PYTHONPATH=src python -m pytest tests

Recent setuptools copies the .so next to src/fastsimdjson.cpp, which is why PYTHONPATH=src is required for the in-place build.

tests/test_loads.py compares loads with json.loads on types and key order; tests/test_lazy.py checks the views returned by parse the same way, tests/test_dumps.py compares dumps with json.dumps and dumpb with orjson (output and exceptions; the orjson comparisons are skipped when orjson is not installed, and recorded orjson results are checked either way), and tests/test_stream.py and tests/test_files.py cover streams and files, and tests/test_minify.py covers minify. The suite covers scalars, integers past 64 bits, UTF-8 strings at every length from 0 to 199, the key cache, random documents, rejected input, deep nesting, padding at a page boundary, a saturated array count, reference counts, and release of a parser that has grown past 64 MB. The corpus test is skipped until simdjson-data is checked out beside the project:

git clone --depth 1 https://github.com/simdjson/simdjson-data.git
pytest tests
# or: JSONDIR=/path/to/jsonexamples pytest tests

The suite builds an ~80 MB document and a list of 16,777,221 integers, so give it some RAM.

The benchmark scripts need simdjson-data and the bench extra (orjson, jiter, msgspec, pysimdjson, cysimdjson). bench.py times loads against json.loads, orjson and jiter, bench_lazy.py times parse against pysimdjson and cysimdjson, bench_dumps.py times dumps against json.dumps and dumpb against orjson and msgspec, bench_many.py times loads_many, and bench_minify.py times minify against parsing and serializing again.

pip:

python -m pip install -e ".[bench]"
python bench.py
python bench_lazy.py
python bench_dumps.py
python bench_many.py
python bench_minify.py

uv:

uv pip install -e ".[bench]"
python bench.py
python bench_lazy.py
python bench_dumps.py
python bench_many.py
python bench_minify.py

Benchmarks

Intel Xeon Gold 6548N (Emerald Rapids), one core unless stated otherwise, Python 3.14.10 (free-threaded Python 3.14.10t for the streams on several threads), fastsimdjson 1.0.0 (simdjson 5.1.1) built with GCC 16.2, the 22 files of simdjson-data. The scripts are in this repository and in the blog repository.

Whole documents

Each parser produces the whole document as Python objects. Speed is the geometric mean over the 22 files (higher is better).

parser	GB/s	vs json.loads
json (standard library)	0.22	1.00×
simplejson 4.3.0	0.23	1.04×
python-rapidjson 1.26	0.24	1.08×
ujson 6.1.0	0.37	1.64×
cysimdjson 26.27	0.43	1.95×
pysimdjson 8.0.0	0.44	1.97×
jiter 0.19.0 (no string cache)	0.44	1.98×
msgspec 0.22.0	0.53	2.38×
jiter 0.19.0 (string cache, the default)	0.54	2.43×
orjson 3.13.0	0.60	2.70×
fastsimdjson loads	0.77	3.46×

fastsimdjson is the fastest on 21 of the 22 files. orjson is 12% faster on numbers.json, an array of floating-point numbers. On numbers, both spend most of their time creating Python floats, which costs the same in both: simdjson parses the numbers of these files faster than orjson's parser (yyjson), but that is a small part of the total. Part of the gain comes from pausing the garbage collector while the objects are built. fastsimdjson's lead over orjson is 1.28× with the collector enabled and 1.18× with it disabled (geometric means); without the collector, orjson is also 3 to 4% faster on canada.json and mesh.json. yyjson 4.0.6 is left out: it returns wrong strings for non-ASCII text. jiter (the parser of pydantic) caches strings by default; turning the cache off (cache_mode=False) makes it slower here.

Parsing is no longer the bottleneck. simdjson alone parses these files at 3.1 GB/s. It accounts for about a third of the time of loads; the rest goes into creating Python objects. Freeing those objects later costs about a sixth of the total. Even if parsing took no time at all, loads would be less than 1.5 times faster.

Parts of documents with parse

If you only need a few values, parse creates only those. Extracting the id and the screen name of the 100 statuses of twitter.json:

method	µs
json.loads	3977
jiter (no string cache)	1602
jiter (string cache)	1156
orjson	971
fastsimdjson loads	844
msgspec (typed Struct)	337
cysimdjson (lazy)	235
pysimdjson (lazy)	169
fastsimdjson parse	156

Here parse is about 25 times faster than json.loads and 5.4 times faster than loads. Most of its time is the simdjson parse itself.

Typed decoding

loads(data, type=T) against msgspec (msgspec.json.Decoder(T) with the same dataclasses, and with the equivalent msgspec.Struct classes), pydantic (TypeAdapter(...).validate_json, with the equivalent models) and cattrs.structure(orjson.loads(data), T). Three cases:

• twitter: twitter.json as a dataclass of 100 statuses, six fields of each status and four of its user, the rest skipped.
• citm: citm_catalog.json typed in full (dicts of dataclasses, lists of nested dataclasses).
• gh: 64 MB of GH Archive events (NDJSON), six fields of each event (loads_many(data, type=T), one call per line for the others).

MB/s, one core of an Intel Xeon Gold 6548N, Python 3.14.10, best of 3 runs of 7 (python bench_typed.py simdjson-data/jsonexamples gharchive.ndjson):

	twitter	citm	gh
fastsimdjson loads(type=)	2597	981	2055
msgspec (Struct)	1520	632	1296
msgspec (dataclass)	1428	467	1156
pydantic 2.14	659	192	514
orjson + cattrs	553	187	420
fastsimdjson loads (dicts, untyped)	933	1074	431

fastsimdjson is 1.5 to 1.7 times as fast as msgspec with Structs, 1.8 to 2.1 times as fast with the same dataclasses, and 3.9 to 5.1 times as fast as pydantic. When most of the document is not needed (twitter, gh), typed decoding is also 2.8 to 4.8 times as fast as building the dicts.

On free-threaded Python 3.14.10t, loads_many(data, type=T, threads=N) read the 886 MB GH Archive file into dataclasses at 1.82 GB/s on one thread, 12.0 GB/s on 8 and 33.3 GB/s on 32.

dumps against json.dumps

dumps returns exactly the str of json.dumps and is 4.2 times faster (geometric mean over the 22 files; from 2.7 times on citm_catalog.json to 10.5 times on canada.json).

dumpb against orjson and msgspec

dumpb, orjson.dumps and msgspec.json.encode produce compact UTF-8 bytes; dumpb and orjson produce identical bytes on every file. With the standard library, the same compact bytes come from json.dumps(obj, separators=(",", ":"), ensure_ascii=False).encode(). Each call encodes a fresh copy of the document (bench_dumps.py), so no encoder gains from what it cached on the objects at a previous call. One core of an Intel Xeon Gold 6548N, Python 3.14.10, GCC 16.2, fastsimdjson 1.0.0, orjson 3.13.0, msgspec 0.22.0. Microseconds:

file	json.dumps(...).encode()	fastsimdjson dumpb	orjson	msgspec
twitter	1804	258	313	387
citm_catalog	3114	651	666	716
github_events	211	21	20	32
gsoc-2018	12153	956	1165	2095
update-center	2647	309	298	514
canada	37821	2743	2480	3978
mesh	8903	876	1049	1401
numbers	2473	201	204	332

Over the 22 files, dumpb is 8% faster than orjson (geometric mean), from 10% slower on canada.json (mostly floating-point numbers) and up to 6% slower on the smallest files (twitter_api_response.json, github_events.json) to 22% faster on gsoc-2018.json, whose strings are mostly not ASCII, and 35% faster on semanticscholar-corpus.json. dumpb is 1.5 times faster than msgspec (orjson 1.4 times) and 8 times faster than json. The comparison was run on a processor with AVX-512, which dumpb and orjson both use to escape strings; other x64 processors use SSE2 and ARM processors NEON.

minify

minify only removes whitespace. The other columns parse the document and serialize it again, which also validates it and rewrites numbers. Geometric mean over the 22 files, one core of an Intel Xeon Gold 6548N, Python 3.14.10, GCC 16.2, orjson 3.13.0 (bench_minify.py):

method	GB/s
fastsimdjson minify	18.64
loads + dumpb	0.54
orjson loads + dumps	0.44
json.loads + json.dumps	0.11

Per file, minify runs from 15.4 GB/s (canada.json) to 20.7 GB/s (apache_builds.json).

Writing NDJSON with dumpb_many

The 593,945 events of two hours of GH Archive (1.87 GB of NDJSON), written back as NDJSON from Python objects. With the GIL (Python 3.14.10), one thread: orjson.dumps on each record, joined, runs at 0.93 GB/s, dumpb on each record at 0.93 GB/s, and dumpb_many at 1.49 GB/s. On free-threaded Python 3.14.10t, GB/s (up to 32 threads on the cores of one socket, 64 on both):

threads	1	8	32	64
dumpb_many(records, threads=N)	1.48	5.6	11.6	10.5

NDJSON to Arrow and back

Two files: 4 million flat records (611 MB, 9 scalar fields), and 304,929 GH Archive events (886 MB), read with a schema of 6 of their fields (two of them structs; the rest of each event is skipped), or inferring the schema. Reading gives a pyarrow.Table (pyarrow.table(fastsimdjson.loads_arrow(data, schema, threads=N))); the time includes the import into pyarrow. Writing starts from that table: fastsimdjson dumpb_arrow, polars write_ndjson and DuckDB COPY ... TO (on tmpfs). Python 3.14.10, pyarrow 26.0, polars 2.1, DuckDB 2.0, fastsimdjson 1.0.0, Intel Xeon Gold 6548N (up to 32 threads on the 32 cores of one socket; 64 threads on both sockets).

All the numbers, in GB/s:

	threads	1	8	32	64
flat, read	fastsimdjson loads_arrow	0.76	5.7	15.0	18.0
	pyarrow read_json	0.18	1.4	5.3	7.8
	polars read_ndjson	0.27	2.0	4.3	3.1
flat, read, inferring the schema	fastsimdjson loads_arrow	0.79	5.6	14.4	12.5
	pyarrow read_json	0.18	1.4	5.0	6.8
	polars read_ndjson	0.09	0.12	0.12	0.12
GH Archive, read	fastsimdjson loads_arrow	1.96	12.9	27.5	24.3
	pyarrow read_json	0.46	3.5	12.1	14.5
	polars read_ndjson	0.54	3.5	4.0	3.1
flat, write	fastsimdjson dumpb_arrow	0.92	3.7	9.6	12.1
	polars	0.45	1.1	1.6	1.5
	DuckDB	0.32	2.3	4.8	3.4
GH Archive, write	fastsimdjson dumpb_arrow	1.08	3.0	4.8	4.2
	polars	0.46	0.52	0.86	1.06
	DuckDB	0.32	1.3	1.1	0.86

load_arrow_file on the same files, from the page cache, reads at 0.69 and 1.5 GB/s on one thread, 9.1 and 13.0 GB/s on 32 threads. The benchmark is THREADS=N python bench_arrow.py flat.ndjson:flat gharchive.ndjson:gh flat.ndjson:infer; path:infer times any NDJSON file without a schema. The flat file comes from simdjson_arrow's benchmarks/gendata.py and the events from GH Archive (curl -sSfL https://data.gharchive.org/2024-03-13-12.json.gz | gunzip > gharchive.ndjson). The charts come from python doc/make_charts.py arrow.

JSON to MessagePack and back

From JSON bytes to MessagePack bytes, and back, against the usual ways in Python, which decode into Python objects, then encode them: ormsgpack.packb(orjson.loads(s)), msgspec.msgpack.encode(msgspec.json.decode(s)) and so on. The 22 files of simdjson-data, geometric means of the speeds, in GB/s of JSON (read when writing MessagePack, written when writing JSON). Python 3.14.10, msgpack 1.2, orjson 3.13, ormsgpack 1.13, msgspec 0.22, Intel Xeon Gold 6548N, one core.

Streams: the 304,929 events of an hour of GH Archive (886 MB of NDJSON), with json_to_msgpack(data, ndjson=True, threads=N) and msgpack_to_json(packed, stream=True, threads=N), against orjson with ormsgpack (each document) and msgpack.Unpacker with orjson (each value), on one thread (one thread per core of one socket up to 32 threads, then both sockets):

threads	1	2	4	8	16	32	64
NDJSON to MessagePack	1.31	2.45	4.7	8.7	14.0	16.0	22.7
MessagePack to NDJSON	1.51	2.26	4.4	6.0	6.0	6.0	5.9

MessagePack to NDJSON stops scaling at about eight threads: one thread must find where the values end (MessagePack has no line ends) before the others can convert them. The benchmark is python bench_msgpack.py (the files of JSONDIR, by default simdjson-data/jsonexamples) and python bench_msgpack.py --ndjson gharchive.ndjson 1 8 32; the charts come from python doc/make_charts.py msgpack.

Streams with loads_many

20 MB of NDJSON (5268 objects and arrays, one per line), made from the same files; best of three runs:

method	ms	GB/s
json.loads on each line	164	0.12
jiter on each line (no string cache)	94	0.21
jiter on each line (string cache)	84	0.23
msgspec decode on each line	75	0.26
orjson on each line	74	0.26
fastsimdjson loads on each line	58	0.34
msgspec decode_lines	55	0.36
fastsimdjson loads_many	49	0.40
fastsimdjson parse_many (views only)	17	1.13

loads_many is 13% faster than msgspec's decode_lines. parse_many only creates the views; reading values from them adds to its time.

Streams on several threads

300 MB of NDJSON (146,624 records: tweets, GitHub events, citm performances, gsoc and update-center entries), free-threaded Python 3.14.10t, collector disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:

threads	1	2	4	8	16	32	64
map_many(len, data, threads=N)	0.90	1.64	2.08	3.53	5.85	8.70	16.8
parse_many(data, threads=N)	1.59	2.78	3.75	4.79	9.01	13.5	20.6
loads_many(data, threads=N)	0.77	1.52	2.29	4.08	6.87	9.99	11.8

map_many is 19 times faster on 64 threads than on one, and parse_many 13 times. loads_many builds every document for the calling thread and is 15 times faster. On 64 threads, these numbers vary by up to a third from one run to the next. With the collector enabled, Python collects after the call returns: it examines the millions of objects just created, which takes about 0.1 s here, with or without threads.

Limitations

• The simdjson parser and the key and string caches are thread-local. release() frees the parser and the cached strings retained by the calling thread and by the worker threads of the parallel streams, and they are freed when a Python thread exits. A parser that grows past 64 MB is freed on its own at the end of that call; its caches stay.
• On free-threaded Python, the worker threads of the parallel streams (threads=N) are started at the first call that needs them and kept until the interpreter exits: starting threads costs about as much as the parsing. Idle workers wait without using the processor. Each one keeps a parser (about a megabyte) and its caches until release(). Free-threaded CPython also keeps the memory of freed objects for reuse, in the thread that allocated it, rather than returning it to the system (the standard json module shows the same): after a large threaded call, the process stays larger, and the next calls reuse that memory.
• Subinterpreters are not supported: the module keeps per-process state (its types, and per-thread parsers and string caches).