BetterJSONStorage 1.0

A tinyDB storage extension using a faster json compiler and compression

Introduction

BetterJSONStorage is a faster ‘Storage Type’ for TinyDB. It uses the faster Orjson library for parsing the JSON and BLOSC for compression. Smaller filesizes result in faster reading and writing (less diskIO), and the extra cost of (de-)compressing has been made up for by using the faster orjson.

An example of how to implement BetterJSONStorage can be found below. Anything else can be found in the TinyDB docs.

Installing BetterJSONStorage

Install BetterJSONStorage from PyPi.

pip install BetterJSONStorage

Usage

context Manager

from tinydb import TinyDB
from BetterJSONStorage import BetterJSONStorage

with TinyDB('/path/to/file.db', storage=BetterJSONStorage) as db:
    db.insert({'int': 1, 'char': 'a'})
    db.insert({'int': 1, 'char': 'b'})

extra

All arguments except for the storage argument are forwarded to the underlying storage. You can use this to pass additional keyword arguments to orjson.dumps(…) method.

For all options see the orjson documentation.

with TinyDB('file.db', option=orjson.OPT_NAIVE_UTC, storage=BetterJSONStorage) as db:

performance

The benchmarks are done on fixtures of real data:

• citm_catalog.json, 1.7MiB, concert data, containing nested dictionaries of strings and arrays of integers, indented.

• canada.json, 2.2MiB, coordinates of the Canadian border in GeoJSON format, containing floats and arrays, indented.

• twitter.json, 631.5KiB, results of a search on Twitter for “一”, containing CJK strings, dictionaries of strings and arrays of dictionaries, indented.

data can be found here.

The exact same code is used for both BetterJSONStorage and the default JSONStorage. BetterJSONStorage is faster in almost* all situations and uses significantly less space on disk.

citm_catalog.json

write_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.1182915	1x
default JSONStorage	0.193683	1.64x

read_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.0098675	1x
default JSONStorage	0.0099165	1x

storage used

storage	used storage in kb	vs. BetterJSONStorage
BetterJSONStorage	83.3	1x
default JSONStorage	540	6.48x

canada.json

write_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.0316401	1x
default JSONStorage	0.0939051	2.97x

read_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.0276127	1x
default JSONStorage	0.057871	2.1x

storage used

storage	used storage in kb	vs. BetterJSONStorage
BetterJSONStorage	1572	1x
default JSONStorage	2150	1.36x

twitter.json

write_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.0104866	1x
default JSONStorage	0.0145437	1.39x

read_speed

storage	time im ms	vs. BetterJSONStorage
BetterJSONStorage	0.0069805	1x
default JSONStorage	0.0078986	1.13x

storage used

storage	used storage in kb	vs. BetterJSONStorage
BetterJSONStorage	155	1x
default JSONStorage	574	3.7x