SplitGrid

Asymmetric split-grid serialization for rectangular numeric and mixed-type grids.

This package was pulled out of WriterAgent (LibreOffice Writer/Calc/Draw AI extension). It is the host flatten + child unpack codec that lived in plugin.scripting.payload_codec, plus the optional Cython flatten accelerator from native/writeragent_vec.

Why this exists

A spreadsheet host often cannot import NumPy: LibreOffice’s embedded Python is a different interpreter from the user’s venv, and loading the user’s C extensions into the host is an ABI footgun. Heavy compute therefore runs in a child process that does have NumPy. The range still has to cross that boundary.

On the wire, a Calc-style range is a nested list: list[list[float | int | str | None]]. Standard pickle walks one heap object per cell. On a 20,000 × 5 grid that is ~12 ms for dump + load. Pack the same numbers into a contiguous float64 buffer and Pickle Protocol 5 moves the bytes in ~0.015 ms; the child can np.frombuffer in ~0.002 ms.

Almost all of the remaining time is host flatten — turning nested Python objects into that buffer without NumPy (~8.3 ms pure Python on that shape; ~3 ms with the Cython helper). SplitGrid is that flatten/unpack. Length-prefixed Pickle 5 framing stays in the application; this package is the codec only.

Column-wise blobs and JSON + Base64 were tried first. Transposing columns on the host builds extra object graphs; Base64 and per-column reconstructs lose the frombuffer win. One row-major float64 buffer plus a sparse string map is faster and simpler.

How it works

The path is asymmetric:

Host pack (stdlib only) flattens a 1D or 2D rectangular grid into a contiguous float64 buffer plus a sparse integer-keyed strings map. Empty cells become NaN. Zip codes like "02138" stay strings — they are never parsed as floats.
Child unpack (NumPy) materializes a pure-numeric grid (strings == {}) with np.frombuffer (ndarray). Mixed grids use a vectorized object-masking path and return nested lists, restoring None for NaN holes.
Host unpack preserves float('nan') from the buffer (does not coerce holes to None). That is the locked egress policy: NaN becomes a spreadsheet error, not a silent blank.

Grids with fewer than 100 cells (BINARY_MIN_CELLS) stay nested Python lists. Force "always" / "never" overrides that threshold (used by A/B tests).

Wire envelope (Pickle5-friendly dict):

{
    "__wa_payload__": "split_grid",
    "dtype": "float64",
    "column_kinds": ["int", "float"],  # per column: int / float / bool
    "shape": [rows, cols],             # or [n] for 1D
    "buffer": b"...",                  # row-major float64 bytes
    "strings": {7: "banana"},          # integer keys, not str(idx)
}

Cell value	`buffer` (float64)	`strings`
`None` (empty cell)	`NaN`	—
`int` / `float`	numeric value	—
`bool`	`0.0` / `1.0`	—
`str` (including `"02138"`)	`NaN`	text by flat index

There is no datetime lane on the float64 buffer. Python datetime objects stringify into strings. Do not add a 'date' column kind.

Jagged 2D grids raise ValueError (spreadsheet ranges are rectangular).

Numbers

Median timings from an asymmetric bench (host = stdlib pack; child = deserialize + materialize).

Ingress, 20,000 × 5 (100k cells):

Format	Pack	Dump	Load	Materialize	Total
JSON nested lists	6.2 ms	22.8 ms	14.6 ms	2.1 ms	45.7 ms
Pickle 5 nested lists	6.1 ms	1.4 ms	2.9 ms	2.0 ms	12.4 ms
Pickle 5 + split-grid	8.3 ms	0.013 ms	0.015 ms	0.002 ms	8.3 ms

Child materialize of a 100 × 100 numeric grid: nested-list pickle then np.array ~0.6 ms; split-grid frombuffer ~0.016 ms. Below 100 cells the envelope is not worth it — that is why BINARY_MIN_CELLS exists. Host pack still dominates large ingress; Cython only speeds that loop.

Python vs Cython

Host flatten is an optimized pure-Python loop:

identity type checks (type(val) is float, val is None)
bound-method capture (buf_append = buf.append)
None as NaN on the fast path
lazy column-state upgrades
rectangular validation before the hot loop

The optional Cython module splitgrid.pack exposes fast_flatten_grid_1d / fast_flatten_grid_2d. It is loaded dynamically and canary-tested at import. If the extension is missing or fails the canary, the codec uses pure Python. Importing splitgrid never requires a compiler. Extension builds use release flags (-O3 -DNDEBUG -g0 on Unix, /O2 /DNDEBUG on Windows).

Install

pip install splitgrid

PyPI wheels include the compiled Cython flatten accelerator. From a source checkout (Cython is built if a compiler is available):

pip install .
pip install -e ".[test]"      # editable + pytest, hypothesis, deal, numpy
pip install -e ".[numpy]"     # child unpack/pack
pip install -e ".[verify]"    # crosshair-tool

Host pack stays NumPy-free. Child child_unpack_* / child_pack_* import NumPy locally.

Test

pytest                  # default: -m "not slow"
pytest -m "not slow"    # same
pytest -m slow          # CrossHair subprocess checks (optional)
make verify             # deal contracts + hypothesis round-trips, no CrossHair

Default pytest does not require the compiled extension. When splitgrid.pack is built, tests/test_cython_parity.py compares Cython and Python flatten components on the same grids.

Bench

Requires NumPy (and the optional Cython extension if you want the accelerated pack row). From a source checkout:

python scripts/bench_serialization.py --direction both
python scripts/bench_serialization.py --child-only
python scripts/bench_unpacking_opt.py
python scripts/profile_pack.py
python scripts/profile_nones.py
python scripts/run_serialization_ab.py --all
make bench

Public API

from splitgrid import (
    BINARY_MIN_CELLS,
    host_pack_split_grid,
    host_unpack_split_grid,
    child_unpack_split_grid,
    child_pack_split_grid,
    host_pack_data,
    host_unpack_data,
    child_unpack_data,
    child_pack_result,
    is_split_grid,
    load_cython_accelerator,
    get_cython_status_info,
)

host_pack_data(..., force="auto"|"always"|"never") chooses split-grid vs nested list. host_pack_multi_data is a thin multi_data wrapper over the same per-grid packing.

License

GPL-3.0-or-later.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

splitgrid-0.1.2.tar.gz (189.4 kB view details)

Uploaded Aug 26, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

splitgrid-0.1.2-cp314-cp314-win_arm64.whl (233.6 kB view details)

Uploaded Aug 26, 2026 CPython 3.14Windows ARM64

splitgrid-0.1.2-cp314-cp314-win_amd64.whl (240.4 kB view details)

Uploaded Aug 26, 2026 CPython 3.14Windows x86-64

splitgrid-0.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (262.3 kB view details)

Uploaded Aug 26, 2026 CPython 3.14manylinux: glibc 2.17+ x86-64

splitgrid-0.1.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (259.3 kB view details)

Uploaded Aug 26, 2026 CPython 3.14manylinux: glibc 2.17+ ARM64

splitgrid-0.1.2-cp314-cp314-macosx_10_15_universal2.whl (316.7 kB view details)

Uploaded Aug 26, 2026 CPython 3.14macOS 10.15+ universal2 (ARM64, x86-64)

splitgrid-0.1.2-cp313-cp313-win_arm64.whl (232.1 kB view details)

Uploaded Aug 26, 2026 CPython 3.13Windows ARM64

splitgrid-0.1.2-cp313-cp313-win_amd64.whl (239.6 kB view details)

Uploaded Aug 26, 2026 CPython 3.13Windows x86-64

splitgrid-0.1.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (261.6 kB view details)

Uploaded Aug 26, 2026 CPython 3.13manylinux: glibc 2.17+ x86-64

splitgrid-0.1.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (258.3 kB view details)

Uploaded Aug 26, 2026 CPython 3.13manylinux: glibc 2.17+ ARM64

splitgrid-0.1.2-cp313-cp313-macosx_10_13_universal2.whl (316.4 kB view details)

Uploaded Aug 26, 2026 CPython 3.13macOS 10.13+ universal2 (ARM64, x86-64)

splitgrid-0.1.2-cp312-cp312-win_arm64.whl (232.2 kB view details)

Uploaded Aug 26, 2026 CPython 3.12Windows ARM64

splitgrid-0.1.2-cp312-cp312-win_amd64.whl (239.9 kB view details)

Uploaded Aug 26, 2026 CPython 3.12Windows x86-64

splitgrid-0.1.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (262.3 kB view details)

Uploaded Aug 26, 2026 CPython 3.12manylinux: glibc 2.17+ x86-64

splitgrid-0.1.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (259.1 kB view details)

Uploaded Aug 26, 2026 CPython 3.12manylinux: glibc 2.17+ ARM64

splitgrid-0.1.2-cp312-cp312-macosx_10_13_universal2.whl (317.0 kB view details)

Uploaded Aug 26, 2026 CPython 3.12macOS 10.13+ universal2 (ARM64, x86-64)

splitgrid-0.1.2-cp311-cp311-win_arm64.whl (232.9 kB view details)

Uploaded Aug 26, 2026 CPython 3.11Windows ARM64

splitgrid-0.1.2-cp311-cp311-win_amd64.whl (238.9 kB view details)

Uploaded Aug 26, 2026 CPython 3.11Windows x86-64

splitgrid-0.1.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (265.2 kB view details)

Uploaded Aug 26, 2026 CPython 3.11manylinux: glibc 2.17+ x86-64

splitgrid-0.1.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (262.2 kB view details)

Uploaded Aug 26, 2026 CPython 3.11manylinux: glibc 2.17+ ARM64

splitgrid-0.1.2-cp311-cp311-macosx_10_9_universal2.whl (315.1 kB view details)

Uploaded Aug 26, 2026 CPython 3.11macOS 10.9+ universal2 (ARM64, x86-64)

File details

Details for the file splitgrid-0.1.2.tar.gz.

File metadata

Download URL: splitgrid-0.1.2.tar.gz
Upload date: Aug 26, 2026
Size: 189.4 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for splitgrid-0.1.2.tar.gz
Algorithm	Hash digest
SHA256	`7cee7a37cdaf31b9c23da09d52b3fa4a9bcc5f632a144af882dbc0c5aba31529`
MD5	`f09bfaada836c065b6e49186258bb451`
BLAKE2b-256	`bca7924627e2e2874b3f3a93bd42cb6a7b12b743de4219a796908e6649ca242c`

Algorithm	Hash digest
SHA256	`d372f6d458ab25d7075c4d091cb42c84e42ddb836cafc7b28ea99f14fe690814`
MD5	`ef85fea405d9646c5be63ea710ac747d`
BLAKE2b-256	`4ff86953735434a6de044eb1d9b5ca3a249e55ed7ddc299f1fd85c7686addb2e`

Algorithm	Hash digest
SHA256	`967897afb2831e2efd3d423d0eceb36e4e361e877eb908d2251bbb866e2b48ea`
MD5	`3ef28fb7249c636f63072d36b1584dab`
BLAKE2b-256	`5fd70dc3c342aa5253089cdc64f8a6496e9e4a9a91b9bac64fe2b5d6ec6264a1`

Algorithm	Hash digest
SHA256	`d6f4b464cb9c28725ebfa81219689691267f9ec1dd23e0c2fc11606bb1c3e2a9`
MD5	`b90f87c071417c6526c194eb110af124`
BLAKE2b-256	`43b190d26f575e9cae77c1c6efc6ac9156993ba3f855763282569612674b9366`

Algorithm	Hash digest
SHA256	`138df313a4c3933281a18b2bdc9fad67a0415ada4873b69d7d7d8fbf4d904ba4`
MD5	`6a5b746fd57c31d7f449921dd337e3da`
BLAKE2b-256	`84b99413133f3d1cbaeb74289e091593ebedd2d3ea78f3ceaa07d3cb069ca039`

Algorithm	Hash digest
SHA256	`3085047c94dfd0758b046629faad98c21a1695f9a0e052f608bd86f82efc0be6`
MD5	`8327dc8d3f988bdf4a5bc6e7aab6cdc0`
BLAKE2b-256	`08e6a52472faa4a32c05b4127297928778ced15789dadd6a2cfd62441caa8f06`

Algorithm	Hash digest
SHA256	`c34cf9bed968e7f79786cabeff056afdb1776b7898cf2b7c47894e0409f6a466`
MD5	`ef5cc7a2e4ab086dde3fa620a569d5f7`
BLAKE2b-256	`3dfaae7e2733febd96ff8c56b75c6a8a589e346a7ffe689e80482fc57576f73a`

Algorithm	Hash digest
SHA256	`62581c99df5fa17bdf38f2a02947d23c3cfebe11cff699db31095f61be59ec8b`
MD5	`e33264b3dddb51151a6e71e0c72350cf`
BLAKE2b-256	`707af59b705823f0c16685e942d9cb96006af87eb61567ed692daca5c40c908d`

Algorithm	Hash digest
SHA256	`1ce292da5fe206c6e07d50e562437e0e5755f84309c5beaa3a32fce0bcd3d537`
MD5	`d4cba9e9ef9a0be03ef7df839e2bcb39`
BLAKE2b-256	`7a1bd1d08c95e3d051811726c39485375a4a1eae56278a8137fea07772fd6e3d`

Algorithm	Hash digest
SHA256	`68df8ab23ec499b54bbd24f334771887bdc6a9fffe06e8f9dc9aa6fbd2970cae`
MD5	`f95b7d27c6036849d412d16997540d62`
BLAKE2b-256	`b76eba737ad303ebd56695d4cf09ccbfb7d11deef4ccfdc37ac37de2cb825a34`

Algorithm	Hash digest
SHA256	`828f5c2036608537414a56f442a2385705f8036ccaed82e4223917db50041574`
MD5	`aba608e3e8e8e8038781be72286a379d`
BLAKE2b-256	`da2eb313c29c35bf3e8de80437d44b20ea6464b2fbc15645c24705157590a421`

Algorithm	Hash digest
SHA256	`984a2c03624fef0e2dfba8d0a6ae2a43ec37cdc3386d19f82e0b62be30c58fa3`
MD5	`369261d2101e6e15fd7dcc57d6468098`
BLAKE2b-256	`a9d8f01b23237e2de984eb1a46d30e5b718a8c03f8c8ec041d418047d9bf78e3`

Algorithm	Hash digest
SHA256	`96117615562f47107d8e725b88233d37580b57e86189db1406d22fc135d45c22`
MD5	`253a9b67b74f34ed2b8bd5db3e81f25b`
BLAKE2b-256	`292d51651478a0118c103115a62c012db13c9e2c2114d3ea964a2c50a6a269c7`

Algorithm	Hash digest
SHA256	`64380cfda87c1690dacb9564aeb2b292b5377e7ee0be3d09ed4c42323bb10cef`
MD5	`a542ce7cf0ebd4794147ff7fc1c2ec8e`
BLAKE2b-256	`efdadc30f6624684ec5e253737bd79c8cfc10607aff55a51d93c102de51c2c26`

Algorithm	Hash digest
SHA256	`1ae2a9cff4441a1e6c9491be6f84936bafd7e05e5dabbd6e45ec71a8ee6a1691`
MD5	`005fed6bca45b28c0c680dbc2f47c397`
BLAKE2b-256	`3e278e5df90af64bc36bdc7e3020c64239d5e84a9cde5cca839ccb315c6647f8`

Algorithm	Hash digest
SHA256	`43e449e6818f7c8fdf553321398bd947bf72236bd19f46bf29ed92180ff85ab4`
MD5	`b131412e2d0255fd580de9b238894460`
BLAKE2b-256	`0b346e842890beb5ccc139a34496ccc6bfabe5b03ada878751b3e3af7bfa8b90`

Algorithm	Hash digest
SHA256	`d8bc122f703a25ab58c3ccb76780448239b92fefe58e286d1ec5162f3d42eb88`
MD5	`82d69233bd98ed9d17e8f1261421ad22`
BLAKE2b-256	`8c8128640107f5b9df49a485e5487cab02b487a79f9de8786950f46146289ed6`

Algorithm	Hash digest
SHA256	`777658feb9cc6509c0ebb82f1ff13f673d73a204f66677c865a6bf285d9a62d3`
MD5	`fd0ea22636cb9afc0393203ce37aa95b`
BLAKE2b-256	`934d7226bac38f53a6cc08e48df4889f0435f0b84e1376960957944148091e07`

Algorithm	Hash digest
SHA256	`88656f8a7dd4eefdf718d714fe70be5425389ed28428b50b6b623d9fcfdaa6d9`
MD5	`1deff759994790d22a0e0c34c525e942`
BLAKE2b-256	`e12f147d4a67f43b876d61f9898e5e27d1df3074b12288960279004a2f516b1a`

Algorithm	Hash digest
SHA256	`d3d339c298789e6803b388112162aa2069333b9c82bbad5338abe23039cddded`
MD5	`c6dbc37b57f2563274ad6fd6312fae5e`
BLAKE2b-256	`86640f718c83d8c0606f36a5a61ee1ef9d48ad4de9469b33a78386921c0c3fbb`

Algorithm	Hash digest
SHA256	`c2fad7eee6c054c4ac8bd0b1984fdf057b1506b92abe4476b42cfdef2ff32a14`
MD5	`169d1f36df30ac2b236803eff3baa1a8`
BLAKE2b-256	`07b620d5aa96a23c427cdef70e9b55d69b613071d5300618975bc5cc5b71ed53`

splitgrid 0.1.2

SplitGrid

Why this exists

How it works

Numbers

Python vs Cython

Install

Test

Bench

Public API

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance