skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 20, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 20, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.2.0-cp314-cp314-manylinux_2_34_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ x86-64

libskene-0.2.0-cp314-cp314-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ ARM64

libskene-0.2.0-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.2.0-cp313-cp313-manylinux_2_34_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ x86-64

libskene-0.2.0-cp313-cp313-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ ARM64

libskene-0.2.0-cp313-cp313-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.2.0-cp312-cp312-manylinux_2_34_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ x86-64

libskene-0.2.0-cp312-cp312-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ ARM64

libskene-0.2.0-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.2.0-cp311-cp311-manylinux_2_34_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ x86-64

libskene-0.2.0-cp311-cp311-manylinux_2_34_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ ARM64

libskene-0.2.0-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.2.0-cp314-cp314-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp314-cp314-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 0f32155686d48c30d166f7bf5a40465e493a85c9aa57cf47e1719a8ab23980c8
MD5 67aa19628459131058db32bde3cfd513
BLAKE2b-256 308a8bcaca7f003715ca604dfcb4e71f97af932db4652b669727a42518c17f89

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp314-cp314-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp314-cp314-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp314-cp314-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 b8218664981610260a6e94adc78c09e253ea7073705b4e368b14b7007e1efea5
MD5 ab08fb9d9202dfdf5c4f741076a7de01
BLAKE2b-256 8c71861a960c07bad80a4c14cdc6effb607c4cfe162e62eed266ceeb1e13c092

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp314-cp314-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f090a6e8b7c2d2ccad66326014c43bb6b8fe3586bdb2a08d7f54550e17de2d06
MD5 b538f0528bd58b4617b7613f71cc54db
BLAKE2b-256 1360a32eee219836241237ebd0ee4ec1ffed7474893f9f1a5da68cff1154a22c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp313-cp313-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp313-cp313-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 db862b44d65bb989c177f526a4d5c59f3936af2cab85321c7ad567d49567afa7
MD5 374a5934ff0117b75cde1698c2a80aa2
BLAKE2b-256 e38873f2edaab56cb33d332eb788c51b0fa4f5141b47baa945abb148ce243ded

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp313-cp313-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp313-cp313-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp313-cp313-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 b301f13d5283998d40ab08c221bc2a555b830ce3a788c0da51333b51ae349200
MD5 6ba7ffb279c638c61e1ebfb92fb1edeb
BLAKE2b-256 1a3a96b8c4d7ce38d48e23ad73575cc69de79e416264ac5331dea2afa59ef3bc

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp313-cp313-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 625ef52370f2101869fc88b09c18f11126dd6220977d9c8d7367c23dfa80dac4
MD5 95281074737cb84372a0a0cb3e7f750c
BLAKE2b-256 30bdb2012c85f212c0798713401e4c58b2a1af8064846c769e2e5e123f32294d

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp312-cp312-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp312-cp312-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 6275512a43aa85ec189d47ffd86bf0a3bf6ded99019be27fdbdee28977401494
MD5 7b9aee2d58782cc157c5fbcaa4a99369
BLAKE2b-256 c6e7228da8271983cca4deaa870226b607b72bb2612a405d488eb6033d828dbc

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp312-cp312-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp312-cp312-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp312-cp312-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 63b9a797f19609c66382bbfe35679cd95c174566f04fe32b52033fc65f6526cf
MD5 fc7461229b97f2d268a71ceb6e34ab08
BLAKE2b-256 daa171aca5bb1255a1415123d312b7b757e8ab159181e94ec3287d5bade0d272

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp312-cp312-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c60d7601cd015e089830bca3a77d3a7e7ddf17d4afc924f6209834c1ce53f747
MD5 f7619fcc50e5f3408ce9a5b05b569175
BLAKE2b-256 096ed557208b61e9de6dcccc233622d5a33316778ea5d79ead4a7037d8c822a8

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp311-cp311-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp311-cp311-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 69db65da7bcb5b3c3f1c7fa908c3fdd8298f1051d52aefde7266a0855f4839ee
MD5 033c6a5c28b0a70a9524c309a5203afb
BLAKE2b-256 297412c77279237b71b4c4c08ff878e5e2ddb2a9d3859bac6751f1ea3e9b7c90

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp311-cp311-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp311-cp311-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp311-cp311-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 c84f86e2d41ba7deb4d77bc736f5be6b65394fad7725ba90e5d0bb11d650c35f
MD5 abdc245313642ad3daef6b5cc2ed9676
BLAKE2b-256 ae98c9b68f99e1058001b1c0b07505ac9ef0f4f9a82ef537c605ef43c433341b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp311-cp311-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2e1331484a60ca0ef8b3c736445bf43c01aa7dc17244d60b75f6332bfaa35e5e
MD5 1b55d980935d0ee4189d2c55513bd22d
BLAKE2b-256 ea2868a0a12de6b0d1a9a800890c51c25aa8a0764cbb7d90fff334c840c301ec

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.0-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

This release

0.2.0 This release

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page