skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 8, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 8, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.5-cp314-cp314-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.5-cp313-cp313-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.5-cp312-cp312-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.5-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 69806009bba5d5dc66c52bbdb50d8dc4b0254f087e405d382602f083314cb137
MD5 95ae4b9b988d96296b65ec2b6ad67420
BLAKE2b-256 d0bde6308c658648c26258be0b19db0844c038b8acf2bdb3913804239f705931

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 e3dd83f8eb780eddd450be8fdace2ef63f1ad56184614436f974c755c0aa47bb
MD5 b58c6f571fa710149302e1e17557b3f7
BLAKE2b-256 3ed8a99a1e6e2667d3e3bf18721d2d280fc44623f759dea6e68ac6439443dfc9

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 af23f8d12aebcfa89af24d55424d0e3885490585725e52b1e8ef9789a5ced36d
MD5 9e55c5f0f010431e6c64b214379e2659
BLAKE2b-256 ab61cb056c09eb9f769057e79dcbd9d08b91c56ef746a4b8bc5ec25285c9d6b4

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 310ef1cd30487467517d48fca532396ba7d6fae2772f35c207ed00a7ed590b1c
MD5 36d3bd3a58bfe011486fd48d8f060dc5
BLAKE2b-256 4b8de258f1abc213288e615efd242db87daeb100e0e3e867a7666e2ac5637856

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 b5838b2f2ebe185324d2ddf8301aa581dbbdc445b3bef7d3acef5ec18e419214
MD5 320e79cfae43eb889e92393f16a61bba
BLAKE2b-256 61dba11915714730eb1f3bf6eeb99c84c76dcfb5163f0d828b9d65365b76476a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 92c884464b758e071857a48c708a11d3f6f0a5467018b7291f58fc8febb9e5b3
MD5 60518d09b5a607774a7306b032f35e13
BLAKE2b-256 491fda1f09b0a5fc7438561d672023aed54fb3a1a7743068e8833579cdc22eb3

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 45c18fd2fbf7d07cd4033806b4b42da1208e04e73058e7319c9b427f80bd8e1f
MD5 15c29a7f448eec0b79bad3d27eea8db4
BLAKE2b-256 39cee5a3713ec9b3c7a44d10829ff52d3711c704d51b9377b9972f661da5ae18

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 177c33083de78ec83809db261c02543442388652a02c5869e34f7deb705a4bf3
MD5 44f588286e5b5a774289de0dc09e7f42
BLAKE2b-256 3b552a3dbb04df2229a13333582952461e37d21eb13550f6b0fa8431c509f57d

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c90153737d5d3375c2b755f594b5ac2251e21214e6e8838d029dd2dde5a30088
MD5 c66a8f54150961e208376ca5eebdee53
BLAKE2b-256 ea02a77427073a1d8d3b5aa1ef933cc31077c3dcee27f030566de1685fe5176f

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 460dd17cd020adedbd200f4e65d77088dd5748dca524ab687af24d8d6c684167
MD5 474be105c213a01cc5cc5eee95e62fce
BLAKE2b-256 e701d1c41978e2256390c8ab161f963096bdc1f9ae2f8310edf246016a9e6c2b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 f391bbc344a981735e24d9e3baeed3023eb8677c1554474a131aaf0f41315335
MD5 59d6b79fecc7f6f01bd9193d2b8dd1f4
BLAKE2b-256 bbd0e68474a044d5874bf736aa5d5337447c73b4b21e2be03d565f4341021e43

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.5-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.5-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c9959ee04ac9e4807ee1a5191801350f8b317a4994e9bc8e189a72126bfe45b2
MD5 fa251da719ac4569e29b3d9e5510b1b6
BLAKE2b-256 8552964b1d8881f56f5d7af71769a68df7c58f88cfe3bf08941816cdad0c7374

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.5-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

This release

0.1.5 This release

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page