🌪️ Vortex

📚 Documentation | 📊 Performance Benchmarks

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

⚡️ Blazing Fast Performance
- 200x faster random access reads (vs. modern Apache Parquet)
- 2-10x faster scans
- 2-10x faster writes
- Similar compression ratios
- Efficient support for wide tables with zero-copy/zero-parse metadata
🔧 Extensible Architecture
- Modeled after Apache DataFusion's extensible approach
- Pluggable encoding system, type system, compression strategy, & layout strategy
- Zero-copy compatibility with Apache Arrow
🗳️ Open Source, Neutral Governance
- A Linux Foundation (LF AI & Data) Project
- Apache-2.0 Licensed
↔️ Integrations
- Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
- Apache Iceberg (coming soon)

🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).

Key Features

Core Capabilities

✨ Logical Types - Clean separation between logical schema and physical layout
🔄 Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
🧩 Extensible Encodings - Pluggable physical layouts with built-in optimizations
📦 Cascading Compression - Support for nested encoding schemes
🚀 High-Performance Computing - Optimized compute kernels for encoded data
📊 Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

Logical Layer: Defines data types and schema
Physical Layer: Handles encoding and storage implementation
Built-in Encodings: Compatible with Apache Arrow's memory format
Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-data

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install latest release
cargo install vortex-tui --locked

# Or build from source
cargo install --path vortex-tui --locked

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

See CONTRIBUTING.md for guidelines.

Reporting Vulnerabilities

If you discovery a security vulnerability, please email vuln-report@vortex.dev.

Trademarks

Acknowledgments 🏆

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

Research in Vortex

BtrBlocks - Efficient columnar compression
FastLanes - High-performance integer compression
FSST - Fast random access string compression
ALP - Adaptive lossless floating-point compression
Procella - YouTube's unified data system
Anyblob - High-performance access to object storage
ClickHouse - Fast analytics for everyone

Vortex in Research

Anyblox - A Framework for Self-Decoding Datasets

Open Source Inspiration

Thanks to all contributors who have shared their knowledge and code with the community! 🚀

Metadata

Release files for vortex-data 0.42.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for vortex-data 0.42.0
File	Interpreter	ABI	Platform	Reset
vortex_data-0.42.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 10.2 MB Jul 24, 2025	CPython 3.10	abi3	Linux glibc 2.17+ x86-64	Details
vortex_data-0.42.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 9.5 MB Jul 24, 2025	CPython 3.10	abi3	Linux glibc 2.17+ ARM64	Details
vortex_data-0.42.0-cp310-abi3-macosx_11_0_arm64.whl 10.3 MB Jul 24, 2025	CPython 3.10	abi3	macOS 11.0+ ARM64	Details
vortex_data-0.42.0-cp310-abi3-macosx_10_12_x86_64.whl 11.0 MB Jul 24, 2025	CPython 3.10	abi3	macOS 10.12+ x86-64	Details

Total release size: 40.9 MB

Release files / vortex_data-0.42.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	vortex_data-0.42.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	10.2 MB
Tags	CPython 3.10 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum How to use checksums	`b58020c968d853a9565ea1bf9e646890b2527abe03d34ba112d91acc718a0001`
BLAKE2b-256 checksum How to use checksums	`1643ba4b14431624043799e012285f85267082149cb1b9e8339583f8d0bbabab`
Upload date	Jul 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.12.9`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vortex-data/vortex
Publishing commit: github.com/vortex-data/vortex/tree/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e

Workflow

Publishing configuration: github.com/vortex-data/vortex/blob/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e/.github/workflows/publish.yml
Publishing logs: github.com/vortex-data/vortex/actions/runs/16503840970/attempts/1

Artifact

Subject: vortex_data-0.42.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
SHA-256 checksum: b58020c968d853a9565ea1bf9e646890b2527abe03d34ba112d91acc718a0001 Verifying attestations

Release files / vortex_data-0.42.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	vortex_data-0.42.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	9.5 MB
Tags	CPython 3.10 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum How to use checksums	`c971188866c51e0ec61ed5c25c43d11bbfa3d1cdd2adfe704a4f2ad4691f5654`
BLAKE2b-256 checksum How to use checksums	`a2d1fb031d7ed01c8157c74b3baab22b1825165ca39ca29b7597963f9e93ecd0`
Upload date	Jul 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.12.9`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vortex-data/vortex
Publishing commit: github.com/vortex-data/vortex/tree/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e

Workflow

Publishing configuration: github.com/vortex-data/vortex/blob/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e/.github/workflows/publish.yml
Publishing logs: github.com/vortex-data/vortex/actions/runs/16503840970/attempts/1

Artifact

Subject: vortex_data-0.42.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
SHA-256 checksum: c971188866c51e0ec61ed5c25c43d11bbfa3d1cdd2adfe704a4f2ad4691f5654 Verifying attestations

Release files / vortex_data-0.42.0-cp310-abi3-macosx_11_0_arm64.whl

Download URL	vortex_data-0.42.0-cp310-abi3-macosx_11_0_arm64.whl
Size	10.3 MB
Tags	CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`679f5478ceffd303c82dc144d25000e0ad2b9b0ae1a3037a1d3daa667a355ff8`
BLAKE2b-256 checksum How to use checksums	`a5a0a050019a4b010f1ef846ea1a82f45512e27fc0b42bfe84f440605e311953`
Upload date	Jul 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.12.9`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vortex-data/vortex
Publishing commit: github.com/vortex-data/vortex/tree/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e

Workflow

Publishing configuration: github.com/vortex-data/vortex/blob/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e/.github/workflows/publish.yml
Publishing logs: github.com/vortex-data/vortex/actions/runs/16503840970/attempts/1

Artifact

Subject: vortex_data-0.42.0-cp310-abi3-macosx_11_0_arm64.whl
SHA-256 checksum: 679f5478ceffd303c82dc144d25000e0ad2b9b0ae1a3037a1d3daa667a355ff8 Verifying attestations

Release files / vortex_data-0.42.0-cp310-abi3-macosx_10_12_x86_64.whl

Download URL	vortex_data-0.42.0-cp310-abi3-macosx_10_12_x86_64.whl
Size	11.0 MB
Tags	CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`b5df657a81bff6004d91987fb53c07a77c7a86ea651f5cf519e428c774c75e45`
BLAKE2b-256 checksum How to use checksums	`26193c98ca286797abbddb64efa3a7415ad5119fba6de987856774b5428740bf`
Upload date	Jul 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.12.9`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vortex-data/vortex
Publishing commit: github.com/vortex-data/vortex/tree/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e

Workflow

Publishing configuration: github.com/vortex-data/vortex/blob/eec0e179bb9c5e808bee6b7677d1cd3f73b8e78e/.github/workflows/publish.yml
Publishing logs: github.com/vortex-data/vortex/actions/runs/16503840970/attempts/1

Artifact

Subject: vortex_data-0.42.0-cp310-abi3-macosx_10_12_x86_64.whl
SHA-256 checksum: b5df657a81bff6004d91987fb53c07a77c7a86ea651f5cf519e428c774c75e45 Verifying attestations