🌪️ Vortex
Join the community on Slack! | Documentation | Performance Benchmarks
If you are interested in closer collaboration, please email info@vortex.dev
Overview
Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:
-
Blazing Fast Performance
- 100x faster random access reads (vs. modern Apache Parquet)
- 10-20x faster scans
- 5x faster writes
- Similar compression ratios
- Efficient support for wide tables with zero-copy/zero-parse metadata
-
Extensible Architecture
- Modeled after Apache DataFusion's extensible approach
- Pluggable encoding system, type system, compression strategy, & layout strategy
- Zero-copy compatibility with Apache Arrow
-
Open Source, Neutral Governance
- A Linux Foundation (LF AI & Data) Project
- Apache-2.0 Licensed
-
Integrations
- Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
- Apache Iceberg (coming soon)
🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).
Key Features
Core Capabilities
- Logical Types - Clean separation between logical schema and physical layout
- Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
- Extensible Encodings - Pluggable physical layouts with built-in optimizations
- Cascading Compression - Support for nested encoding schemes
- High-Performance Computing - Optimized compute kernels for encoded data
- Rich Statistics - Lazy-loaded summary statistics for optimization
Technical Architecture
Logical vs Physical Design
Vortex strictly separates logical and physical concerns:
- Logical Layer: Defines data types and schema
- Physical Layer: Handles encoding and storage implementation
- Built-in Encodings: Compatible with Apache Arrow's memory format
- Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)
Quick Start
Installation
Rust Crate
All features are exported through the main vortex crate.
cargo add vortex
Python Package
uv add vortex-data
Command Line UI (vx)
For browsing the structure of Vortex files, you can use the vx command-line tool.
# Install pre-built binary (fast, recommended)
cargo binstall vortex-tui
# Or build from source
cargo install vortex-tui --locked
# Or run via Python without installing
uvx --from vortex-data vx --help
# Usage
vx browse <file>
Development Setup
Prerequisites (macOS)
# Optional but recommended dependencies
brew install flatbuffers protobuf # For .fbs and .proto files
brew install duckdb # For benchmarks
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup
# Initialize submodules
git submodule update --init --recursive
# Setup dependencies with uv
uv sync --all-packages
See the development workflows for Python binding and documentation development, including Maturin rebuilds, targeted tests, and documentation checks.
rust-toolchain.toml pins the toolchain used for development and CI, and is kept on the latest
stable release. Building Vortex as a dependency only requires a toolchain that satisfies the
Rust version compatibility policy.
Benchmarking
Use vx-bench to run benchmarks comparing engines (DataFusion, DuckDB) and formats (Parquet, Vortex):
# Install the benchmark orchestrator
uv tool install "bench_orchestrator @ ./bench-orchestrator/"
# Run TPC-H benchmarks
vx-bench run tpch --engine datafusion,duckdb --format parquet,vortex
# Compare results
vx-bench compare --run latest
See bench-orchestrator/README.md for full documentation.
Performance Optimization
For optimal performance, we suggest using MiMalloc:
#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;
Project Information
Rust Version Compatibility Policy
The policy: Vortex supports the four most recent stable minor releases. Writing the latest
stable release as 1.N, that means 1.N, 1.N-1, 1.N-2, and 1.N-3 all build Vortex, so the
three minor releases older than the latest stable release. Only the minor version is constrained;
Minimum Supported Rust Version (MSRV) declared in Cargo.toml must be no newer than 1.N-3,
patch releases are never a factor.
An MSRV older than 1.N-3 is always acceptable — supporting extra releases cannot break the
guarantee. An MSRV newer than 1.N-3 does not meet the policy. For example, once 1.98 is the
latest stable release:
| Declared MSRV | Status |
|---|---|
1.94 or older |
Acceptable — supports more releases than required |
1.95 |
Exactly on policy |
1.96 or newer |
Does not meet the policy |
The MSRV is raised in occasional deliberate steps rather than on every Rust release, so it drifts relative to that bound.
How the policy is applied:
- The MSRV is declared once, as
rust-versionin the rootCargo.toml, and inherited by every crate in the workspace. That value, not this document, is the source of truth. - The toolchain pinned in
rust-toolchain.tomltracks the latest stable release and is independent of the MSRV. It is what contributors and most CI jobs build with. - CI enforces the declared MSRV in the
Rust (MSRV)job, which builds the publishable crates with exactly that toolchain. When it fails, the first choices are to express the code without the newer Rust feature, or to hold back the dependency update that raised the requirement. Raisingrust-versionis a last resort.
License
Licensed under the Apache License, Version 2.0.
Governance
Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.
Contributing
Please do read CONTRIBUTING.md before you contribute.
Reporting Vulnerabilities
If you discover a security vulnerability, please email security@vortex.dev. See SECURITY.md for our full security policy, including emergency reporting and CRA stewardship.
Trademarks
Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see https://lfprojects.org
Acknowledgments
The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.
Research in Vortex
- BtrBlocks - Efficient columnar compression
- FastLanes & FastLanes on GPU - High-performance integer compression
- FSST - Fast random access string compression
- ALP & G-ALP - Adaptive lossless floating-point compression
- Procella - YouTube's unified data system
- Anyblob - High-performance access to object storage
- ClickHouse - Fast analytics for everyone
- MonetDB/X100 - Hyper-Pipelining Query Execution
- Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Format for the Many-Core Age
- The FastLanes File Format - Expression Operators
Vortex in Research
Open Source Inspiration
- Apache Arrow
- Apache DataFusion
- parquet2 by Jorge Leitao
- DuckDB
- Velox & Nimble
Thanks to all contributors who have shared their knowledge and code with the community! 🚀
1 maintainer
Metadata
1 maintainer
Release files for vortex-data 0.87.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| vortex_data-0.87.0-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details |
| vortex_data-0.87.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| vortex_data-0.87.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.11 | abi3 | Linux glibc 2.17+ ARM64 | Details |
| vortex_data-0.87.0-cp311-abi3-macosx_11_0_arm64.whl | CPython 3.11 | abi3 | macOS 11.0+ ARM64 | Details |
| vortex_data-0.87.0-cp311-abi3-macosx_10_12_x86_64.whl | CPython 3.11 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 221.9 MB
Release files / vortex_data-0.87.0-cp311-abi3-win_amd64.whl
| Download URL | vortex_data-0.87.0-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 46.6 MB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
3029695a580e0bb24d42ec318b6f1b804ec9e914e899bd315374796aa2459d0e
|
|
BLAKE2b-256 checksum How to use checksums |
fd30a0e4191aa3fbb6fd204081a4fab226591b0709e29f2265bfe5d556c62f64
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / vortex_data-0.87.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | vortex_data-0.87.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 44.0 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
9525588a85777613b84b058e850fb34bded51d5da094eebfa392cdf693a7f2ea
|
|
BLAKE2b-256 checksum How to use checksums |
28ad07c86129bcec928153c1f815fd308884804173582fdfb8d7d1876eb0db01
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / vortex_data-0.87.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | vortex_data-0.87.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 42.1 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
8d84444cc3b85cfb688b7faa2336b4d316ec1413e787d13cb8b7920cc45e5bbc
|
|
BLAKE2b-256 checksum How to use checksums |
d2af396ba55210030c3536999ad773ac27ece47c97fca035fd6de9251f5f6907
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / vortex_data-0.87.0-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | vortex_data-0.87.0-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 43.8 MB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
2be8ca67e6b33d199bbf2d3139c49583ca769c674479741dd2d9838741e15ba1
|
|
BLAKE2b-256 checksum How to use checksums |
42d4460990777a570c7120cec54dfc53cf14c1ee4457d0169a904df48646d99f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / vortex_data-0.87.0-cp311-abi3-macosx_10_12_x86_64.whl
| Download URL | vortex_data-0.87.0-cp311-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 45.4 MB |
| Tags | CPython 3.11 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
9764d77d02696494d038d7c547bc7c345e86f6e1a282c1404bb360cd074cb078
|
|
BLAKE2b-256 checksum How to use checksums |
830800540cf5657f514f6f6d07535fdf2dfeaca5c2a880ad4fe60346727bdebe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency log
