🌪️ Vortex

Build Status OpenSSF Best Practices Documentation CodSpeed Badge Crates.io PyPI - Version Maven - Version

📚 Documentation | 📊 Performance Benchmarks

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

  • ⚡️ Blazing Fast Performance

    • 200x faster random access reads (vs. modern Apache Parquet)
    • 2-10x faster scans
    • 2-10x faster writes
    • Similar compression ratios
    • Efficient support for wide tables with zero-copy/zero-parse metadata

  • 🔧 Extensible Architecture

    • Modeled after Apache DataFusion's extensible approach
    • Pluggable encoding system, type system, compression strategy, & layout strategy
    • Zero-copy compatibility with Apache Arrow

  • 🗳️ Open Source, Neutral Governance

    • A Linux Foundation (LF AI & Data) Project
    • Apache-2.0 Licensed

  • ↔️ Integrations

    • Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
    • Apache Iceberg (coming soon)

🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).

Key Features

Core Capabilities

  • ✨ Logical Types - Clean separation between logical schema and physical layout
  • 🔄 Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
  • 🧩 Extensible Encodings - Pluggable physical layouts with built-in optimizations
  • 📦 Cascading Compression - Support for nested encoding schemes
  • 🚀 High-Performance Computing - Optimized compute kernels for encoded data
  • 📊 Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

  • Logical Layer: Defines data types and schema
  • Physical Layer: Handles encoding and storage implementation
  • Built-in Encodings: Compatible with Apache Arrow's memory format
  • Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-data

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install latest release
cargo install vortex-tui --locked

# Or build from source
cargo install --path vortex-tui --locked

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

See CONTRIBUTING.md for guidelines.

Reporting Vulnerabilities

If you discovery a security vulnerability, please email vuln-report@vortex.dev.

Trademarks

Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see https://lfprojects.org

Acknowledgments 🏆

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

Research in Vortex

  • BtrBlocks - Efficient columnar compression
  • FastLanes - High-performance integer compression
  • FSST - Fast random access string compression
  • ALP - Adaptive lossless floating-point compression
  • Procella - YouTube's unified data system
  • Anyblob - High-performance access to object storage
  • ClickHouse - Fast analytics for everyone

Vortex in Research

  • Anyblox - A Framework for Self-Decoding Datasets

Open Source Inspiration

Thanks to all contributors who have shared their knowledge and code with the community! 🚀

Project links

Data verified by PyPI on Jul 24, 2025
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.86.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ngates from gravatar.com ngates

Credits

Author: Vortex Authors <hello@vortex.dev>

GitHub Statistics

Data verified by PyPI on Jul 24, 2025
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache-2.0)

Requires

Python >=3.11

Provides Extra

polars pandas numpy duckdb

Tags

vortex

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

vortex_data-0.42.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

vortex_data-0.42.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (9.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

vortex_data-0.42.1-cp310-abi3-macosx_11_0_arm64.whl (10.3 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

vortex_data-0.42.1-cp310-abi3-macosx_10_12_x86_64.whl (11.0 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file vortex_data-0.42.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for vortex_data-0.42.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b6a2471ad7043de9ed19fe556ceb68b526561102965d1a9365706434767a130a
MD5 d5344664883a78275670fb96f18c394d
BLAKE2b-256 0c59afa1d3c4d70ee6dba3b2258d60342aaeb22da615a1d41a2efa0ff81b4611

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.42.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.42.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for vortex_data-0.42.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 138b86999ae02c9f87a46fb84f435016ad69399204d8ea37d1cf029ff5844387
MD5 cfacf7b1a55bed009aa6c0d98b8dc96e
BLAKE2b-256 4083d9637880afb65db35ceaca5a69f71a46181fa27020a0c2173657bb50c27f

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.42.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.42.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for vortex_data-0.42.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b62121c917d3dde013a79a696a8c87553e4c92a0dc5c3b0d0960a465cd88d337
MD5 84771fa336717a72387ed51571846195
BLAKE2b-256 5d492d4dea664e801359cf1796739174b2d9c59be7ff3d50860a82570634ca23

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.42.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file vortex_data-0.42.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for vortex_data-0.42.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 f7bfc1f301dd1c40dd4537683056835cc4e50bbec8ff22175e1ac64a636b49a4
MD5 12ea6ca64eb8db05d83f39ed68a18189
BLAKE2b-256 a96c21e684c278562e6dda78797f5fa2b2802a9f31c87494b963fee030d87a47

See more details on using hashes here.

Provenance

The following attestation bundles were made for vortex_data-0.42.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: publish.yml on vortex-data/vortex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.86.1

5 files

0.85.0

4 files

0.84.0

4 files

0.83.0

4 files

0.82.0

4 files

0.81.0

4 files

0.80.0

4 files

0.79.0

4 files

0.78.0

4 files

0.77.0

4 files

0.76.0

4 files

0.75.0

4 files

0.74.0

4 files

0.73.0

4 files

0.72.0

4 files

0.71.0

4 files

0.70.0

4 files

0.69.0

4 files

0.68.0

4 files

0.67.0

4 files

0.66.0

4 files

0.65.0

4 files

0.64.0

4 files

Yanked

0.63.0

4 files

Yanked reason: #6975

0.62.0

4 files

0.61.0

4 files

0.60.0

4 files

0.59.4

4 files

0.59.3

4 files

0.59.2

4 files

0.58.0

4 files

0.57.2

4 files

Pre-release

0.57.0rc0

4 files

0.56.0

4 files

0.55.0

4 files

0.54.0

4 files

0.53.0

4 files

0.52.1

4 files

0.52.0

4 files

0.51.0

4 files

0.50.0

4 files

0.49.0

4 files

0.48.0

4 files

0.47.0

4 files

0.46.1

4 files

0.46.0

4 files

0.45.0

4 files

0.44.1

4 files

0.44.0

4 files

0.43.0

4 files

This release

0.42.1 This release

4 files

0.42.0

4 files

0.41.2

4 files

0.40.0

4 files

0.39.0

4 files

0.38.0

4 files

0.37.2

4 files

0.37.1

4 files

0.36.1

4 files

0.36.0

4 files

0.35.0

4 files

0.34.0

4 files

0.33.2

4 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page