🌪️ Vortex

📚 Documentation | 📊 Performance Benchmarks

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data analytics. It provides:

⚡️ Blazing Fast Performance
- 100-200x faster random access reads than Apache Parquet
- 2-10x faster scans with similar compression ratios and write throughput
- Efficient support for wide tables with zero-copy/zero-parse metadata
🔧 Extensible Architecture
- Modeled after Apache DataFusion's extensible approach
- Pluggable encoding system
- Zero-copy compatibility with Apache Arrow

🚧 Development Status: This project is under active development. APIs and file formats may change, and some features are still being implemented.

Key Features

Core Capabilities

✨ Logical Types - Clean separation between logical schema and physical layout
🔄 Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
🧩 Extensible Encodings - Pluggable physical layouts with built-in optimizations
📦 Cascading Compression - Support for nested encoding schemes
🚀 High-Performance Computing - Optimized compute kernels for encoded data
📊 Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

Logical Layer: Defines data types and schema
Physical Layer: Handles encoding and storage implementation
Built-in Encodings: Compatible with Apache Arrow's memory format
Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-array

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install latest release
cargo install vortex-tui --locked

# Or build from source
cargo install --path vortex-tui --locked

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

See CONTRIBUTING.md for guidelines.

Reporting Vulnerabilities

If you discovery a security vulnerability, please email vuln-report@vortex.dev.

Trademarks

Acknowledgments 🏆

This project builds upon groundbreaking work from the academic and open-source communities:

Key Research Papers

BtrBlocks - Efficient columnar compression
FastLanes - High-performance integer compression
FSST - Fast random access string compression
ALP - Adaptive lossless floating-point compression
Procella - YouTube's unified data system
Cloud Object Storage Analytics - High-performance analytics
ClickHouse - Fast analytics for everyone

Open Source Inspiration

Thanks to all contributors who have shared their knowledge and code with the community! 🚀

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

vortex_data-0.34.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB view details)

Uploaded May 19, 2025 CPython 3.10+manylinux: glibc 2.17+ x86-64

vortex_data-0.34.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (8.9 MB view details)

Uploaded May 19, 2025 CPython 3.10+manylinux: glibc 2.17+ ARM64

vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl (9.6 MB view details)

Uploaded May 19, 2025 CPython 3.10+macOS 11.0+ ARM64

vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl (10.4 MB view details)

Uploaded May 19, 2025 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file vortex_data-0.34.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: vortex_data-0.34.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: May 19, 2025
Size: 9.6 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for vortex_data-0.34.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`afb7b5342ae13300025683eba35c43f9e7ff32e48e2eda5b01687ec1b6362464`
MD5	`fd2850c25aed2e52a8f3a6af6fab3c2a`
BLAKE2b-256	`5fab24c1967dba6c1dac7ae9416f393a2c8116a6e29af8fbce11dccea4b1b291`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.34.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vortex_data-0.34.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Subject digest: afb7b5342ae13300025683eba35c43f9e7ff32e48e2eda5b01687ec1b6362464
- Sigstore transparency entry: 215204427
- Sigstore integration time: May 19, 2025
Source repository:
- Permalink: vortex-data/vortex@4645a228ef0d381283478043cfb2e13edcd529ed
- Branch / Tag: refs/tags/0.34.0
- Owner: https://github.com/vortex-data
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@4645a228ef0d381283478043cfb2e13edcd529ed
- Trigger Event: release

File details

Details for the file vortex_data-0.34.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: vortex_data-0.34.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: May 19, 2025
Size: 8.9 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for vortex_data-0.34.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`20043e60c76d31fc2a2f40a1a1480e296c229fe233824575bbc89ff92f39fbbf`
MD5	`16d9c4487706df4d8e5984495f1f926e`
BLAKE2b-256	`186e47f0cb47edc495626d758d89e0d4e0f00d62c5ceec89caf5547d51b3b57b`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.34.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vortex_data-0.34.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
- Subject digest: 20043e60c76d31fc2a2f40a1a1480e296c229fe233824575bbc89ff92f39fbbf
- Sigstore transparency entry: 215204430
- Sigstore integration time: May 19, 2025
Source repository:
- Permalink: vortex-data/vortex@4645a228ef0d381283478043cfb2e13edcd529ed
- Branch / Tag: refs/tags/0.34.0
- Owner: https://github.com/vortex-data
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@4645a228ef0d381283478043cfb2e13edcd529ed
- Trigger Event: release

File details

Details for the file vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl
Upload date: May 19, 2025
Size: 9.6 MB
Tags: CPython 3.10+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`b5b37a792fd1a1d6ac4d98a866b947b27df62205c17160fb4f60e1974dc27dd6`
MD5	`42a413c6371a907492381393563c4ce7`
BLAKE2b-256	`1aa0b7b0eb37cb425af7404fb6a9dfa05e7eca0ab75f5a9a4bf1afa1549991d3`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vortex_data-0.34.0-cp310-abi3-macosx_11_0_arm64.whl
- Subject digest: b5b37a792fd1a1d6ac4d98a866b947b27df62205c17160fb4f60e1974dc27dd6
- Sigstore transparency entry: 215204432
- Sigstore integration time: May 19, 2025
Source repository:
- Permalink: vortex-data/vortex@4645a228ef0d381283478043cfb2e13edcd529ed
- Branch / Tag: refs/tags/0.34.0
- Owner: https://github.com/vortex-data
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@4645a228ef0d381283478043cfb2e13edcd529ed
- Trigger Event: release

File details

Details for the file vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl
Upload date: May 19, 2025
Size: 10.4 MB
Tags: CPython 3.10+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`f6781c3fe1d4fb4ba6350c7a7c0afe20cff7fbe51c2b5550ce9fae6df2940e55`
MD5	`9dd88e93a68aa7defb4ecd75a41ae6fd`
BLAKE2b-256	`8d42bef4818a9c940cd2efb5e82c2c43130b71e955981a7ea2c58b16f178a94c`

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: vortex_data-0.34.0-cp310-abi3-macosx_10_12_x86_64.whl
- Subject digest: f6781c3fe1d4fb4ba6350c7a7c0afe20cff7fbe51c2b5550ce9fae6df2940e55
- Sigstore transparency entry: 215204434
- Sigstore integration time: May 19, 2025
Source repository:
- Permalink: vortex-data/vortex@4645a228ef0d381283478043cfb2e13edcd529ed
- Branch / Tag: refs/tags/0.34.0
- Owner: https://github.com/vortex-data
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@4645a228ef0d381283478043cfb2e13edcd529ed
- Trigger Event: release