skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 14, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 14, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.9-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.9-cp313-cp313-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.9-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.9-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 c858c23c7c226e8c194cb2f04b72e2ad9587b770077bc0cb8241f8376efdd164
MD5 1ee42bd408412122ab4746153f5682f1
BLAKE2b-256 9435ab7e23386162a92efd72dac255879c76fc3c3fc215f7cfd6a5bd396dbee7

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 3ca751669aeee709abbe44da015cc6feb0d382e7bf14d6373e611a90d11160c3
MD5 c0c35fe087b38785d2111774fb83ff35
BLAKE2b-256 aadf01e17af5ac81c41cd6f0d874a72553a71965445503e91896107ac812fafc

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2369c5282c6c32e1b357e1c22eb08e638aaa98cd86aed3b80b90f0af10dca86a
MD5 d7c9943b532a0410678cf208c8ce4258
BLAKE2b-256 a8b4b406b8f5b6fe194ac674c90b68789417c242049ff8331db2970075f73e72

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 e4605738e200335018cffc5818b93724621ebac14b30ca597a77a50b507abf7a
MD5 13ae57115b3422e8d7f2882fb434668d
BLAKE2b-256 c710e9582a3b65db1ca607b99841a36ad1262eed4c154000ad2e809471051b06

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 97bf900efaa014bef961e9e51f3e72307b1bbd68a265be45e0af151f2d4de6f8
MD5 e270729a7dfa61862698f99754b2ed28
BLAKE2b-256 b443429d9eec8ff7e2f46847f46c824c541a0aefbccb0d08af5ad17dec188ecf

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1700b73dd88120b8a8d2b7c7e505c63874f89f5c3cee50ceec6568d6feb37818
MD5 e11a449c9fc35bd4a425447920ab7eb5
BLAKE2b-256 0a328df28558bc7e7a02025c12d3673b1dfe4a409533dc4ab7faba610aee93c1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 4f4f4f56fbf92a7074514cd4e17ce52fd433b611ed47288081315de43100d600
MD5 3f53f393a284e6e1b73bc3f19ad223f4
BLAKE2b-256 04c7b36eb9197e84be6c89d9e02a6416ff56a6b702333c9f45c7b064c94c6710

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 79a261e7cbc89c053c13e7343debfd7fb2493a7c2e3305ec480794aea54ceaab
MD5 145bd2bbbab4317b3d21a76965befa69
BLAKE2b-256 3567db2fe1448ba2355fabe9e2b18080a38cab487065875b804598b704e7c3fd

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 890bd41915088042828a5958df7cc2de12794d761d1b6e97fe3614ce4ad51d04
MD5 b00d88cc18ee235afef7581734157da7
BLAKE2b-256 90c91f251f2f2343189ca8a2802f696cfbdc0fc8a55b3bc89942035c32cd391c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 3097a0636c68b5bc4669902615a79079399034ba228c13fa790b3d0ec3d75793
MD5 acd828eb3cbc4d9c16ddee61433c1aa9
BLAKE2b-256 73ec46c90e23a147967e212001a4d5ff04c7ed6ddee72ae0ceadafc23b173965

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 dabb39f8101af2043f1ce32e39ee3c751715d667a79ca471bbeb6019bc120548
MD5 d85a5e9ad641ad674b084ea6fafcd45b
BLAKE2b-256 37261cd5d229ef2c969a7d5e387167dc808088c87f665344e819b93e3290dd93

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.9-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.9-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a73f89a37c93790e31d45019216e4ab60aa6619967fc360d9fc3716706d3fff0
MD5 bc4ab0b8709225fc1d5e0e328d4d4855
BLAKE2b-256 26501676e206e25a145fe4e41b43d13e73fcb5b90cdc4665dd4e44b8c110b779

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.9-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

This release

0.1.9 This release

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page