skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 18, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 18, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.10-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.10-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.2 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.10-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.10-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.10-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.2 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.10-cp313-cp313-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.10-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.10-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.10-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.10-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.10-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.10-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 833805b7be5b2778f62ad7c9ed4943adcb0b7b2d7547dc8eb92ddb5b0c3e7462
MD5 8e819eb7657e7af91e04177fa8986207
BLAKE2b-256 c1b4ed436c4a2e1ab0fd204dc1bdc9d28c765e6d4a44f7849d4fe590f56d9365

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 a40583b4fb3ef6f0ecedf6d501c3ca7488b1eb90b61314be2d9334e74d03d4a2
MD5 eeb02bd090b395b7fc7f2b48dbefcf75
BLAKE2b-256 a4ebcc4e024650a38adc991aadfe43e9fa5f706fdd4939b34fa2ae2e9d03e084

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c7c287351952cf0955483a5be470bce9cca7d3e8f281858f459514a4e4f56f1f
MD5 cd60f5a65c7be9bc52faf9cc51eac298
BLAKE2b-256 f69eddd7ce126dea305c5c2fa3379729b4c54dfd5b242fb644eb1bd054402522

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 7f0f4574b908612f4d67951a4200301f020a3b4a719a5b3ab1813b3678eff590
MD5 066faa1905e06735f4b498a71a618e56
BLAKE2b-256 54b32773432de8763941eea569e9dd7b1c7c2f5ab99951d19e611a6862a4e08c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 414c2c24052ceb36bb8ecd47b8fe41c4b43544d1f83115e6dd2740fda277035b
MD5 58f77391f2ca4fcc9a6082ed5dc277cd
BLAKE2b-256 497006187c7d39f0a1456dbcd89e0c79149b0a3ba546aee15f4be061d1656fc6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3775c1d73364dd2562046eb0cc043b49ca19bcc3a8a56a9314d2641d3a4ed3e5
MD5 5bf5132b0cd8d757b9c64122160dd300
BLAKE2b-256 e2e2ecddc8bd8bb62491709eeaaba4b8869e6d806711c485f331d2b30d59329d

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 2b8c0af7a39c91c7fcc85b0606126d2ac1755d8d2f3259fec84a3d10c0ff3edc
MD5 52b511c7cc58e6ad00681fb48911f79a
BLAKE2b-256 8632405451524938af4740c5006c43f45d6d2138b28bbcea19f4bf8051582ecd

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 bdfc5c88b328afbdb7a97ec0bb3eb416057e08641848e93b30cf12fd87b14ac6
MD5 f1ea09422b1d4f3460eb4eadf16d6ccd
BLAKE2b-256 947000cbc557b775798dc797e23ac6fed76d97668ef85077781453d9c22f26d9

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c5c4256111046341113e7690aaef1a7c1cab32add419c77d6a27aab4b0425e90
MD5 b8d864d0f8113b57b49db077e4cfb7fb
BLAKE2b-256 0ca01c188b546fc8ca71e8eb4edfd66619ef2220738fdcbadc6bb04260ae1b5d

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 a0b6d14526f5897be5ecbf4d89ed6291a68a98252478faa59f483a4a07a5f8a9
MD5 514b0ace69b73488cb838e74b89f6a94
BLAKE2b-256 a1ff5d17b24c0ce64e3215664555010e5c1b4760ffe8f5860ed7bb1fd0232b6a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 88d6be78b493093ce0d0c7779332dc5db781d2e1ab29dbc49a2e9fb0747b0eea
MD5 c0c02ef8de2c6316aa1f21660601bf56
BLAKE2b-256 141e7496bb3744800e21378fc2a8821c955f4a54fea8ed4fd2e0811991518327

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.10-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.10-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0caeb3fa23664a71b3211a241a47754baa8663579dc355d93c7c1a832860e07d
MD5 d41b7c4ebf423b0588370a46e62c7d7e
BLAKE2b-256 2191bab0a83988541c51b74a4d8546385c150cee0f17ada379c163e2040c2957

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.10-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

This release

0.1.10 This release

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page