skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 20, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 20, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.11-cp314-cp314-manylinux_2_34_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ x86-64

libskene-0.1.11-cp314-cp314-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ ARM64

libskene-0.1.11-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.11-cp313-cp313-manylinux_2_34_x86_64.whl (4.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ x86-64

libskene-0.1.11-cp313-cp313-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ ARM64

libskene-0.1.11-cp313-cp313-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.11-cp312-cp312-manylinux_2_34_x86_64.whl (4.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ x86-64

libskene-0.1.11-cp312-cp312-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ ARM64

libskene-0.1.11-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.11-cp311-cp311-manylinux_2_34_x86_64.whl (4.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ x86-64

libskene-0.1.11-cp311-cp311-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ ARM64

libskene-0.1.11-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.11-cp314-cp314-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp314-cp314-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 57faa979cfe386703c9e53bf61811639845cf675de19175158e55fe31c4f7f39
MD5 627a1d2e742228f82f6e928205769ec7
BLAKE2b-256 f7eeea9df3a21ccadca3be33cba66a8584a38ef52cbdf4d1fe5fdf9a5bf02fb5

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp314-cp314-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp314-cp314-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp314-cp314-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 7ea12ccebab31b64e3940c9f98e3030f348668c098bd6c6806f4df8c954c14c4
MD5 f4f3e7d9aa77f3dcee5d764d3bc2b7f7
BLAKE2b-256 fbcd5643e2c6ad80b6e1a0c3d273569f581075f407563e30c2de288bfa01fb70

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp314-cp314-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e937c05f4d747ee88e0dd250b27417efe89851d03a178ad900ae02d724e52756
MD5 618b87b3985f8448942b84814c00355c
BLAKE2b-256 83256f284689f6da78c9d06043fdc632f67ccc0e0d92c577325269a97b210208

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp313-cp313-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp313-cp313-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 2f7038a6602b95210f585b22b38bb8343f38c35c876475e71849389d496a25f4
MD5 0c70da63ab6ec5bf4301e2bf63da189a
BLAKE2b-256 02c727378ba936b7656c50520f5697162992220bf86b4e7e0168adb934fceca6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp313-cp313-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp313-cp313-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp313-cp313-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 3e3a3d898f9948ed003442a7484949cafc8b9343e9f26e4468ae1662f316edb2
MD5 3793ee133b19e810a98145e43e6dd08e
BLAKE2b-256 6cd90c18d135ec57f92268048b497854b8746c3b0869d89b5b3cae178c06e250

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp313-cp313-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 314b392f005594b35a97d66aeb83db337eb40f2b3e16b46f86072fc864b8ceec
MD5 d725d005dc414ffa5063ccb401c1cc7f
BLAKE2b-256 5a182c102f3180f16c76dcc8478d1bf2613a99fe212edeb104805747c1d560c6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp312-cp312-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp312-cp312-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 8619962a90f840b9ead57b3f832f4d7715d1456ba01768edef0866f0cb16fef1
MD5 90d4cd4eb94cc89277809ed7703712ef
BLAKE2b-256 a310c0d29a042252787bfad2c8a8eac2a0c5a15a203edfa17b093f9443faa83a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp312-cp312-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp312-cp312-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp312-cp312-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 11efb06f5719209be2ede13f418560c64de5aceae1f07f0694d2f827d346b343
MD5 9f5329aba4bbde893e7e5a0fcd96f3bd
BLAKE2b-256 d6c9a0cb37b804c362d2673799463e494b19664d9c6886cb315bd8e5b6672e19

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp312-cp312-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 12576e37c216e674f6676f22c2d0dca160417a775b7a9a26e79acedddb748a23
MD5 bc9e3a9bb7c4830140587eede300f945
BLAKE2b-256 2cb38a9bd74a2c4b61bdb59c826b2e2a773e3f09aefabc2984f1cf0e70887f5e

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp311-cp311-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp311-cp311-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 8823a91d7f581f772f344c79ed1574feddfa9991dbbc18bef37951b4a71a6734
MD5 4f615c2b57036e804da057471139c9a1
BLAKE2b-256 c5eb9a80b1e4935fc0e245f8d97bef85eb37b03b841b5eea8096b4b5c85c8f16

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp311-cp311-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp311-cp311-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp311-cp311-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 58d591787c8f3616d66fdc7521c134b9cc8b84dcf9197bff92c2323e67c0c031
MD5 d461ac339f1014cf7b35ff77c2a75a5d
BLAKE2b-256 e50521ab6ff85179d8c821b25cb2c522e4aadb44d323a401186bf350ef652cb6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp311-cp311-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.11-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.11-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9cc1e594a4bc86a2bd625d3a8bed312a2de3875978110dad253736782107ce09
MD5 b63ee00d14d29b9f56d511bd281f8fd7
BLAKE2b-256 4054db876533f43d4bbd00d61cf636edf861b45e66a96ae8a3d633b7a9b96bb2

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.11-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

This release

0.1.11 This release

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page