skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 26, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 26, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.2.1-cp314-cp314-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ x86-64

libskene-0.2.1-cp314-cp314-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ ARM64

libskene-0.2.1-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.2.1-cp313-cp313-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ x86-64

libskene-0.2.1-cp313-cp313-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ ARM64

libskene-0.2.1-cp313-cp313-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.2.1-cp312-cp312-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ x86-64

libskene-0.2.1-cp312-cp312-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ ARM64

libskene-0.2.1-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.2.1-cp311-cp311-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ x86-64

libskene-0.2.1-cp311-cp311-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ ARM64

libskene-0.2.1-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.2.1-cp314-cp314-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp314-cp314-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 4a878993614186912b9a6e2ba1c1e566991238d8d925c1d203ac4b5355d6f51f
MD5 4d699a45a56d5d0b9bc78e77a4122ade
BLAKE2b-256 97f0c9195fbd80c82583966d00db6ae78f29f3b14c71fbdc5132e6ae6a23f6c1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp314-cp314-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp314-cp314-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp314-cp314-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 75d1cdc6636469c5e09215e968b950df79991099c1c5041eee8d4158c8d3520c
MD5 34843a688d127a744c6cb987174b45c7
BLAKE2b-256 7c482d1a1102a44b31c6e7529e92fe760e2660a5594e825610ca89a671a9fce1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp314-cp314-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a797e3bdaa9e91d04fd64b011dd0ffa8d2a10982edf14666b1f0a9df3c886f92
MD5 e5efb20313b1866b56c2685043f70af7
BLAKE2b-256 f34748643cbf4c8a216de91797ab6006744261b8ea1a08ad0ba9c0d29373a3d8

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp313-cp313-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp313-cp313-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 b4e41d46091e3a2cf03614fdd7f01823add3dc93e3fbcde3f3d736c32ea42b1d
MD5 ab36a9bcefa02f3b65cc638d2d1907e6
BLAKE2b-256 addb62c09843dbcfb3f88aeb8c95c0daf894495668d7bec3243f308f93ecc3b4

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp313-cp313-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp313-cp313-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp313-cp313-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 624966c3520f0b6f936b5a4c1d3df0ed0b75a8fb302c3efcdc2d6d07884f4644
MD5 8ce406eb10a82e99ad335535eca38398
BLAKE2b-256 e8d62262371c34c4da3b6c2ebd4019057926a70cb2315abc92fbf10b04d93e44

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp313-cp313-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 32eb67a181d1dd9217b914c06e044621e13b0282014815743ca5c588580dfa42
MD5 90a0284e1fc162b52bd4534b49f0cd35
BLAKE2b-256 6fccbab09071d69f2f2779c818ed06b48ed8488021815892be4455ff73f2f0e4

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp312-cp312-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp312-cp312-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 9aea9ca110e16b14d990000c7d007638dfb045dcc234599de400b9cfbc97348b
MD5 708cff3ca1d0eb0fff31dedd3b2f57f5
BLAKE2b-256 1b6a454dfd8ef891ef83c2cb011d086f232bb9a4b36d881c1d08c81553f4679d

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp312-cp312-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp312-cp312-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp312-cp312-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 46c8647855f72eddc96a0cd83de481ca5d3ae2a1900282f6d5121e39d2dc3122
MD5 13567e752a7d0f5f948aec8973482b9d
BLAKE2b-256 5b25f241b43f13834cbd8aa460cd6b6dd7487c2acd4da34e574e08bfd6c9e5e9

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp312-cp312-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3ddccb6df0f241f22b63d50ec4d482827e9e618eaa67afae58fb22ff11de19b3
MD5 045e5b2b0b0b72ed22010593faad34fa
BLAKE2b-256 387dd3e8fb6b6fba0c64b0407eac58d9df01c66edb3cf50058dddeff02242ec7

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp311-cp311-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp311-cp311-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 b334707f774a3148cb6ce64987aa7774b4b88d920420497ea5075b3c9e36caab
MD5 dc5a2244f2b862bb058cb69144d0e18b
BLAKE2b-256 5a43d7da197163ff6c908457ebb042b54c224fad84d4458fd154d1654b4d2c2e

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp311-cp311-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp311-cp311-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp311-cp311-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 30ab1e3c9b911654140d0ededcecdda1e6531642deaa57a5d9f9d1ad37e42bbb
MD5 476cb271a45ace2fac0bc892c5df6edc
BLAKE2b-256 0a69a378868b79aaca283f928eb458057b6ea55d952696078b3c88fdfa865551

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp311-cp311-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 7906f3ded2a443ba2d8d16d08f8dd42995fe0211d7000aa5202bd0bac79ff59e
MD5 2c43713f3d14a3c6c9c6ac4a48d440fd
BLAKE2b-256 f59bee1b2c8258b6bec71d2de2f287d88befacffb5dc2039b27e95f6e09a847b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.1-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

This release

0.2.1 This release

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page