skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 10, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 10, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.1.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

libskene-0.1.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ ARM64

libskene-0.1.7-cp314-cp314-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.1.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

libskene-0.1.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

libskene-0.1.7-cp313-cp313-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.1.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

libskene-0.1.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

libskene-0.1.7-cp312-cp312-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.1.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

libskene-0.1.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

libskene-0.1.7-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.1.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 c1700620a276ee62eb7968c6c1d2cb53e0446af6a0dcc987ba9ba220063da169
MD5 b8998bcdd7fe7e648818e03afeed707a
BLAKE2b-256 ea6ecb0b19d1a8100094ab259a82a84a782a7ebb49163eededf1f14bb3cac389

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 9c228fd5495e2f1f2e17c17b935ae0905723ec66f82fcc41524da7e0efe9d4a2
MD5 1e26a334ae594387607ba78be76e268f
BLAKE2b-256 68294b27418ad1b1cc0564582d56ea8dcf875c9d039dd8d1bba6e45cf56150b1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 69ef35a7e2948de086bd65fc4c9d58171894ca67319ca664211e5bf8ff96c73c
MD5 357689cc58fe32059d0373bb07b456f6
BLAKE2b-256 56c36f9469ec2813a23b6442de0cd54f01655b4aa10461c0016f24c412624cac

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 522c1ddadc6c0f226dc13b6e15e7ad9245d07f10348dd6a4ae45ea22c846f63b
MD5 b8fd7a24c76c2b9f078db439b3764b3f
BLAKE2b-256 7c9775400a0a022d3999dc3c8df94f28a377b5b5050787737fb923c359c04453

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 0351f3f10695b02761ef01a81dacb914580a8de074a26f6da4c03760e1cb4851
MD5 65690e7f9c4c188b3c4a4d675334166e
BLAKE2b-256 50753c110367675ec8fa8660d44b1aa500bcf7bec59e62fde26bd0e7ee8a2e0b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e9cfb13317e69bc5e9d163d9d6ba26b1f420359edc1a107bca91e00aac135f56
MD5 a561a84c3f9169b53f5960977cace166
BLAKE2b-256 f1c122e3ef4b66f75bc4e5ac9bdef5dbc80062e6138f4d10241d43b92977f6cc

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 3264538c5aca7f52f94cc2d8e4a89bac03cd358a2eeb1b18971a306568785c2a
MD5 768bc13ba86c830c3fe335ac24cc76ec
BLAKE2b-256 8f05a8e4e01c5e571fdb3504234e5e8a0faec74ec3ca23affc8f951429b83037

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 1faf20745a4eb002195647800c451052f58f778583101c256a9eb5566dc3fbdf
MD5 a0660dadcd957d65ed8184502746d8a1
BLAKE2b-256 560010693fd974b22532d9908e197077cc901de057bbf8dc53ff38dddc7e9c9b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2951f0727f0a4b2a685eb70df6f12d7d30b673d95c9daf83305f86acb5d5baba
MD5 d247df6aeb4f6f6be1fa8a12076cdbe8
BLAKE2b-256 32404eb326cabc237cd0492ab0eb7f4a238491a423732ae04acd998054c4abc5

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 b8d9bcce7048e0ba81c5a45346dd8f3f52e1d81b3a7b3b5be92409561760282d
MD5 f31d36fe5eacf4a79340a9ad5afcac63
BLAKE2b-256 fb7fdc77311c75117440549197f322a857cb207025404aaa02a2a8654ee472cb

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 83f29e479cf596ba73b36e42420e5f139e4cc3e5adf7b3e11a9b7b06d41d7fcf
MD5 871d8943fe0a05c474141b70904c5ad4
BLAKE2b-256 d5deba7855ac15536c5d0f544ee5092d927a73fe2dd1ff2af295b33fd6410d6f

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.1.7-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.1.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 92cd43ffe82692a2294edeea034cc868b4b699649203f28f3e83b6aa5f3e39b9
MD5 26fe6bbaeff11eab206c29bb91e22860
BLAKE2b-256 e06dfbde461aacb8197a6809438a61b762f82c771d61a4f6e5457a09ae6be324

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.1.7-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.2

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

This release

0.1.7 This release

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page