skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one row group of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: three encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and per-section zstd), zone maps, and bloom filters.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 7, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 7, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.0 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.0-cp314-cp314-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.0 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.0-cp313-cp313-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.0 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.0-cp312-cp312-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.0 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.0-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 efd3d7567ebe982592339e8b842065feb27a8a5c784839edb4e67317acf2628f
MD5 976ff67e9e9761bd657ad0a52d150259
BLAKE2b-256 708e91431bb8129c58d9e0b4bb6ddb47aae8f07f282cd67f6acfb311ad97fc50

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 8ec75cbd2cdd383f497d80280200b1ca16fd68cc76a1ca5cbcf3f2d4f426de17
MD5 4ca0f66ed3373257a1ea303ee4b7a55d
BLAKE2b-256 2bf0ac5519a21cd405962c6ccc137de7e799c00882bbba286d31ff1c00d235d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 744c088bcf81f39d6dd5c71fcf5988e757c8c5a3eff11b5aa1c42703b97f3046
MD5 b3048f94bc5b2220e6377c9613c7b778
BLAKE2b-256 2331f6ef87b5b2bbe986e4f5506af5f1393489b3be1457cc829a38e800131e20

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 af0f3becaa6dca7273fd9b90c4a23f1732446da90cc4aa1708696ef98da4c27f
MD5 1cafebb9599b7106b27646413ed8fd5a
BLAKE2b-256 4bc505def40c3c2ae70f5b5dfbe84990c3e6e018884e7df74cb511e4c6c9258c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 290a9320b67fe036c7095dcbb2b011f33f306434c9e57e2aa75a4f0885201f24
MD5 55c24a5b8c7eb79b620cadb92c0e1e9f
BLAKE2b-256 975a8137cabea5c0a2ff39092a2c344b0a3f71a937fc3dddbf2ebfdc0cb892f0

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 07214411aa3c97e0d0dbfb9c38be095cc637bcf68f8fca8a2ae1548f62eccee8
MD5 1fceab53d758199c6d9aa8c76e5b7a00
BLAKE2b-256 c036a2e2763a6f85f27379613f5959023dc84ec1426989e62deffcc0ca1373b7

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 baa9a967ea259e4dcf337111f7ca46c8ab2e7a4de4584317036b6079f7f7d11d
MD5 5f3a71bcec75294719eb7aa21720b198
BLAKE2b-256 b3f3393e10242e844c006f7468e646ecae03c42c2fba299ea12d087ee93adca3

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 8688e1a73dbbcbfece48bb56a6abe62242c25af70c6aee081b527937ad7770b6
MD5 3cb4f3fbb8873228b2d817edd95d7d6c
BLAKE2b-256 a1e0c878756ead25527f829f72d04fbccc025c66da04cf8158f73d1bb87bcbf8

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0f58c0930e3f1c9857e5d0821cadb2bafbfb0e16133fea23cab8ba7c29cb0cb2
MD5 a6ca93b85f51dbea9fcaa27872dd8435
BLAKE2b-256 64a0e30c396eabe80ec36a55816bc54e6219ab96eaba0d01db3270f2c8e21c4a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 827f6593b88d64fb5a1079e4008ff196afc67e6c5c3f8714bbd417f78c9a46cb
MD5 3f3c48b691a128cbc1eb279c208f6c8c
BLAKE2b-256 a04e41294e8d47d94d29b6c77fa261ac4012d9ca0e8b31869d8f10e5f4925c4c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 d768e1db2470d7d2327ade5fa12aec63aa4ba388efaa52a6fe808ab016fc1561
MD5 0409523c6c8b17ebc6783f67bd58e982
BLAKE2b-256 3c8cbe13eb81c0ec874ea78559c947e2a8397d58740fcc70b376a4af56780ff6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 7ea230e3cf1f3060cde1ce91b2c964308d88138d80f4e220f38da40f9cef29bd
MD5 b3b62f7768b8e7499d313e472a6e9dd2
BLAKE2b-256 79cb8120b70fd1172ba1f417048b6cf429289587bfd0131ea30d66185dd5d0c2

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.0-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

This release

0.1.0 This release

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page