skene

A draken-native columnar file format: reader, writer, and specification.

.skene stores one or more row groups of draken vectors losslessly — including the things Parquet drops. An IPv4 column is a UINT32 refined by an IPV4 logical descriptor; Parquet stores the 32 bits and loses the refinement on every round trip. .skene carries it. It likewise restores a column's dictionary encoding and layout hints rather than re-deriving them.

Reads are per row group and the index is always explicit — read_morsel(buf, row_group). Pruning goes through the small file-level index (read_metadata), which carries the schema, the row group directory and every row group's per-column statistics without opening a single row group's own footer.

It is not portable, and no foreign reader is promised. Parquet stays the default for interchange and for most stored datasets; .skene is for the cases where the draken-native round trip is what matters — query results, sort spill, and datasets we want optimised rather than interoperable.

  • FORMAT.md — the normative specification. Complete enough to write an independent reader from.
  • Rationale — opteryx-core/docs/SKENE_FILE_FORMAT_DESIGN.md: why Parquet and the existing IPC format were rejected, what was traded, what is still open.

C++17, no dependencies, no Python.

Build

skene imports draken's headers; it never copies them. Point it at an opteryx-core checkout — a sibling directory by default:

make            # library and tests
make test       # build and run every test
make DRAKEN_ROOT=/path/to/opteryx-core

Status

v1 is a DRAFT and is not frozen. Byte layouts are implemented and tested, but fields may still change without a version bump until v1 is released.

Implemented, writer and reader: the complete required-section layout in AS_WRITTEN mode — every family (fixed-width, BOOL, the string family including length-only columns, ARRAY with recursive children, NULL), all three selection kinds, LogicalType round-trip, per-section and footer checksums, head/tail framing, column selection, metadata-only reads with per-column byte extents, and version identification. That is exactly the shape sort spill needs.

Also implemented: value ordering (sort + deduplicate each column's values, with the selection carrying row order, so a predicate becomes a binary search and data_length is the exact distinct count) and statistics (min/max ordinals, null count, exact 128-bit sum).

Also implemented: four encodings (bit packing for selection codes, delta+bitpack for ascending integer data, and two per-section general-purpose codecs — zstd and lz4), zone maps, and bloom filters.

The two codecs are postures rather than alternatives to pick per section: zstd for ratio (7.34x at level 9 on ClickBench section bytes), lz4 for read latency (4.49x, but decoding at roughly the rate the reader's own uncompressed path runs at). zstd's decode rate does not vary with the level that produced the bytes, so a low zstd level gives up ratio and buys nothing back.

Per-section compression matters more than expected: on TPC-H, skene without it is 1.9-3.8x larger than the equivalent ZSTD Parquet, and with it 0.92-1.09x. See BENCHMARKS.md.

Not implemented: permutations — deferred because nothing produces a stored sort order yet — and migration, which needs a v2 to migrate from.

A file never claims more than was computed: a column ineligible for ordering is written as-written, and a statistic that cannot be defined is absent rather than zero.

Versioning

A build reads two versions — the one it writes and its predecessor — and writes exactly one. Older files are migrated forward one hop at a time using retained release binaries: binary vX migrates (X-1) → X, so a v1 file reaches v4 by running v2, then v3, then v4. See FORMAT.md §12.

Because of that, any build can identify any file even when it cannot read it — which is what freezes the first six bytes of the header for all time.

Project links

Data verified by PyPI on Aug 27, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author: Justin Joyce

GitHub Statistics

Data verified by PyPI on Aug 27, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

libskene-0.2.2-cp314-cp314-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ x86-64

libskene-0.2.2-cp314-cp314-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.34+ ARM64

libskene-0.2.2-cp314-cp314-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

libskene-0.2.2-cp313-cp313-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ x86-64

libskene-0.2.2-cp313-cp313-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.34+ ARM64

libskene-0.2.2-cp313-cp313-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

libskene-0.2.2-cp312-cp312-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ x86-64

libskene-0.2.2-cp312-cp312-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.34+ ARM64

libskene-0.2.2-cp312-cp312-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

libskene-0.2.2-cp311-cp311-manylinux_2_34_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ x86-64

libskene-0.2.2-cp311-cp311-manylinux_2_34_aarch64.whl (3.7 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.34+ ARM64

libskene-0.2.2-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file libskene-0.2.2-cp314-cp314-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp314-cp314-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 59c3134169e0aba530efad3bc974ca61b640c93dfe95ee4edd54e7121802d99a
MD5 592f3018b2ff145a86a6404478969f40
BLAKE2b-256 79a07da124a370c98230d4d1316ff283dbd4c2853370127a05c8e489fd561fd8

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp314-cp314-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp314-cp314-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp314-cp314-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 f7d4f6c8223f8d30242819276aa9be624a323137269a9ac898cbd0d440a33d08
MD5 d399b1b6db02e6a4e34104739092d83c
BLAKE2b-256 c6853dd1b9f35d7493b76361daf720315f6547400956d9f4a71d6f91be388d9a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp314-cp314-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c0716dde6f6f932e33ebc941083f59a450e416916d2628fd494089a0e3acb12b
MD5 a81af4112acf1a9fb48d591cc6e9b1b4
BLAKE2b-256 5e5ce172d0fd8054eeea0335fcb2c36430de8df209ca9bdcd99b1e7ea777568c

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp314-cp314-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp313-cp313-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp313-cp313-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 ffd014f0bab55d9296fb641a1d3e073c5226bd437df213cce08b27190642fab6
MD5 5d3d162644ae83fbc3ef869d547b79ba
BLAKE2b-256 5eb6af34b3ea4bb8b3a0c41217e175d24750976ec25edeee3e948f5599448245

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp313-cp313-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp313-cp313-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp313-cp313-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 05200d3bf83a11e01d68ead6d0cb6b2ffa8c5452b954824cc5a927cf4b05ec81
MD5 27c4f2ed8d6088bdf7e54ceba4ff908a
BLAKE2b-256 c3f35a40416b7add30fc0b0f829068d7e0819bb6a7d6e3ca8e0b835209b3b55b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp313-cp313-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ca7eba0f44090911ef87f5ee1747d334925f9fe5fcd0df3272816f3157fd0a44
MD5 d86eef9e9e05f5e0994467bba4fe46fd
BLAKE2b-256 829eaa67b583ac1daf5e4bf3ab6d5ce920336fb753d66f5d1ce5c32d911ae026

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp312-cp312-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp312-cp312-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 7b3bc1c72eb50d6cbb6c9c57810283357794d24f8e5185b3beaa5fd8023b3687
MD5 c34fd7d909eb0494702cbfbccf4ad1ab
BLAKE2b-256 cfb8c1fe1bceb07e84bbf885cd84b032b328054d61128639b6e59a3aa44f98c6

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp312-cp312-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp312-cp312-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp312-cp312-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 a334ada41d424adf2457b5b36fc624a969f3ce68decd2b28b4e50c396de1ce4c
MD5 7f81b5ac8c5e73c552c6758079b6c459
BLAKE2b-256 263edc7749ff38e642b705446d10dd20a23f0c13ad86be867d6270e0e31092d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp312-cp312-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 cbd9e5931418e383668f1c332e9ae2a9bd6b92c8bb743fa08bf287da71cfbcf6
MD5 22fa9ed3fae85d8db58f189e2aeddd4a
BLAKE2b-256 df498daa80eef575ec55e1687e4a90e51ef10d3d700208c772a248650d51509a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp311-cp311-manylinux_2_34_x86_64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp311-cp311-manylinux_2_34_x86_64.whl
Algorithm Hash digest
SHA256 bc99dfc5976abb9f7acd2cc750af31402db90dc8cbc0648ab7828afd7b451f5b
MD5 68313b95fb2c7bc21c9a9d9ecb171b28
BLAKE2b-256 d0962708fa7ee7b904ae91cb9a80c862ca1578f6191d348d21e7952eccb23700

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp311-cp311-manylinux_2_34_x86_64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp311-cp311-manylinux_2_34_aarch64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp311-cp311-manylinux_2_34_aarch64.whl
Algorithm Hash digest
SHA256 8594405988e54c434e374c783dab94e4b2eb3fd3b6725a6a9ec1fe1120848bd0
MD5 c05717726d3edb281c1f571695da42ea
BLAKE2b-256 8aab53e4c18217737c7697f8cae93d40a928d82d703def725e8404bd6f7dbf9b

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp311-cp311-manylinux_2_34_aarch64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file libskene-0.2.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for libskene-0.2.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e3fe77864b4d64126d2136685b30c42f2b6fc9583b0e0c168e5437189d9ec181
MD5 f23056d0c167ec0925e815a897f6846f
BLAKE2b-256 2acf0f6f9c481af2e4605fa47998c8cb647d2cf6df947923e594cacf44a9074a

See more details on using hashes here.

Provenance

The following attestation bundles were made for libskene-0.2.2-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release-skene.yaml on mabel-dev/opteryx-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.2.2 This release

12 files

0.2.1

12 files

0.2.0

12 files

0.1.11

12 files

0.1.10

12 files

0.1.9

12 files

0.1.8

12 files

0.1.7

12 files

0.1.6

12 files

0.1.5

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page