🌪️ Vortex

Build Status OpenSSF Best Practices Documentation CodSpeed Badge Crates.io PyPI - Version Maven - Version codecov Cite

Join the community on Slack! | Documentation | Performance Benchmarks

If you are interested in closer collaboration, please email info@vortex.dev

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

  • Blazing Fast Performance

    • 100x faster random access reads (vs. modern Apache Parquet)
    • 10-20x faster scans
    • 5x faster writes
    • Similar compression ratios
    • Efficient support for wide tables with zero-copy/zero-parse metadata

  • Extensible Architecture

    • Modeled after Apache DataFusion's extensible approach
    • Pluggable encoding system, type system, compression strategy, & layout strategy
    • Zero-copy compatibility with Apache Arrow

  • Open Source, Neutral Governance

    • A Linux Foundation (LF AI & Data) Project
    • Apache-2.0 Licensed

  • Integrations

    • Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
    • Apache Iceberg (coming soon)

🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).

Key Features

Core Capabilities

  • Logical Types - Clean separation between logical schema and physical layout
  • Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
  • Extensible Encodings - Pluggable physical layouts with built-in optimizations
  • Cascading Compression - Support for nested encoding schemes
  • High-Performance Computing - Optimized compute kernels for encoded data
  • Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

  • Logical Layer: Defines data types and schema
  • Physical Layer: Handles encoding and storage implementation
  • Built-in Encodings: Compatible with Apache Arrow's memory format
  • Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-data

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install pre-built binary (fast, recommended)
cargo binstall vortex-tui

# Or build from source
cargo install vortex-tui --locked

# Or run via Python without installing
uvx --from vortex-data vx --help

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

See the development workflows for Python binding and documentation development, including Maturin rebuilds, targeted tests, and documentation checks.

Benchmarking

Use vx-bench to run benchmarks comparing engines (DataFusion, DuckDB) and formats (Parquet, Vortex):

# Install the benchmark orchestrator
uv tool install "bench_orchestrator @ ./bench-orchestrator/"

# Run TPC-H benchmarks
vx-bench run tpch --engine datafusion,duckdb --format parquet,vortex

# Compare results
vx-bench compare --run latest

See bench-orchestrator/README.md for full documentation.

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

Please do read CONTRIBUTING.md before you contribute.

Reporting Vulnerabilities

If you discover a security vulnerability, please email vuln-report@vortex.dev.

Trademarks

Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see https://lfprojects.org

Acknowledgments

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

Research in Vortex

Vortex in Research

  • Anyblox - A Framework for Self-Decoding Datasets
  • F3 - Open-Source Data File Format for the Future

Open Source Inspiration

Thanks to all contributors who have shared their knowledge and code with the community! 🚀

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ngates from gravatar.com ngates

Credits

Author: Vortex Authors <hello@vortex.dev>

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python >=3.11

Provides Extra

cuda polars pandas numpy duckdb ray hf

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.7 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.17+ x86-64

vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (40.5 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.17+ ARM64

vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl (43.7 MB view details)

Uploaded CPython 3.11+macOS 11.0+ ARM64

vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl (45.5 MB view details)

Uploaded CPython 3.11+macOS 10.12+ x86-64

File details

Details for the file vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 214bb09d361b20ba3841d91eb06c6cc977b38bcfb5b685c644195daa202e70c0
MD5 09949b5eff09c6b098b2423cadd1e83a
BLAKE2b-256 1e8cbae50237f59d1ef3f14d11c9faea23a45b997fa475f36d47902ffe384b5d

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 37975ca830d9424033a02f3ae22d38aa14c4510db2a7f48750fb9290aa917d4c
MD5 cb316dcfa8f56eb205bce69d4dd7512b
BLAKE2b-256 406f46ace4ea10f225b30925f152eaef01e487da9355f5c830ee0fcf3e5923c3

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2c8b449de8e7d4d26b1d53012c3717a7bd09865e97f3d546565710c5915ed6d8
MD5 5a42cb72069cefa26ce3d9b77a155366
BLAKE2b-256 330755ae88e4a9ca81885213f476f037a0f6cf203ba6a25789149b5a331f7a10

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 18ab3ff6172d941df919cf7d2c74fb157431a73a44354c1133d51013b551e443
MD5 3dd1a0eac0736f8f3ef8a759e90c0577
BLAKE2b-256 89f8d94dfbbbd6a342b5b0bab475fd49548df8ded5e9eb8c7ec14f700cd91f04

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page