skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.8-cp314-cp314-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.8-cp313-cp313-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.8-cp312-cp312-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.8-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 f07c8eae7d5d956dca8c8a2cab0ea73d2418a8392f85fd3ed50b4ea7fe9e6a67
MD5 1ffacf4d722ac93b802fa9fee940a16f
BLAKE2b-256 b9f1b0dd689f252e675c4f55921d8e6157775a11c35bfdf12736c0d12c3e2ca5

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 16046846aaed9aaccda57fac37fc317d0e5fdfd2527c6d5d4c4496b5d3238c3a
MD5 eca16507f1b6f97f9062fd5d8699b8e6
BLAKE2b-256 d527304e143f5f0208eb7eba22711db93d254306d0c296797ac318c2f8796dbe

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c4614d0308946788bc59844ef9c8553d7dc7d1fb7ec45f29faa46d150c4c6fbf
MD5 57248d6a89272da543f933631a76ae38
BLAKE2b-256 1819a855d888688e09b82a69dd20620b7445f77aa8400480d6ee81be04e044ae

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 e15edf13df419fc0922aca9280a89a1c43542d6f1e86440fb39b53c9a87aa3ce
MD5 67c5eb9c8ca145c3c7497a764d8a18a4
BLAKE2b-256 94b30eaa0612c47ac52ea930864de760380530319359cb2e6978c3d4d739f4aa

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 2360c0ec9ebc1b8824ea0c9ba5d38adc14c1ef526fb0115654b6f788c8a786e5
MD5 4b9c575284b23b989bda8bf7a13b0753
BLAKE2b-256 ff8b0d9dd19891ddc879d64e784178abc62d92d6080e28ea8a45d425c12a16ee

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 057ee3b7d7a3a748ef82a240557610b5a022f6c5abb53780ca9a480292612f54
MD5 cb356249e94c08fbdc4e24b6e65c5f74
BLAKE2b-256 2994122c1c611310be266b2b17fb358b5284745a97b19d47801f196022eafe08

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 0ae6d02539ff41c67353b1879e0f433b98fedb1bf1528195f2918e07c951d003
MD5 4985c343c4b3b14f443ca5e5cb70cc3a
BLAKE2b-256 5631098c7030944449444dd1d0aad4b74174fcbc9d6321a27b60b41b1febc6ce

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 71cd95bb8017906a3b4256c075d86a84a55ee0b17616ebacf802ae29e81e72dd
MD5 8e22054efb13e798ac75cafc7077cfc2
BLAKE2b-256 3d61cabfae5f0404c9f1c0413633c453dfc62d395304a7ba2beed6a86b4905fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 232cb8bd4f782ef337a3b766599aeee323920eea23d20ed011cdf5ddac56abfc
MD5 14e3f8aed2614508eaf389fd05bf2bfa
BLAKE2b-256 215999532167deda21ca9b8465d71f2bc67a5500bb635e05e8c4379aa0024063

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 bf47aab1b43c78a093d9eb97472f995a7b2baebf03ecb14e937f8aead9f82801
MD5 28577453b9ceaa5e77cfb03ef389e2ac
BLAKE2b-256 767cdcc1ac2d5fa7793277a528dc649bec4d12bb802cd8d6c736d4318ba39950

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 e33134cd1f4ca6c9c837fd840f029c90d08e6ad538a5e2f82f522da4f7af9dfc
MD5 7ee01c9fab794c59222fe39c088ab46a
BLAKE2b-256 e9f198e8b37b0deb636513921d3bb44d8fff8e8e5445ee8a6d3f6c7e500b24d2

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.8-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 10d7291a68ef82dd4b563bc63d38255f901a39cd7a7a6d110a249a402ecf87ea
MD5 d1320ec3f8e43abc88b71841e36a90b0
BLAKE2b-256 74bebfef6239a397766572762ac008b9a6d53f9e8579f531bf4feeed1e5a4c86

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.8-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

This release

0.1.8 This release

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page