Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Out-of-Core DataFrames to visualize and explore big tabular datasets

Project description

Documentation

What is Vaex?

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. It calculates statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid for more than a billion (10^9) samples/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

Key features

Instant opening of Huge data files (memory mapping)

HDF5 and Apache Arrow supported.

opening1a

opening1b

Read the documentation on how to efficiently convert your data from CSV files, Pandas DataFrames, or other sources.

Lazy streaming from S3 supported in combination with memory mapping.

opening1c

Expression system

Don't waste memory or time with feature engineering, we (lazily) transform your data when needed.

expression

Out-of-core DataFrame

Filtering and evaluating expressions will not waste memory by making copies; the data is kept untouched on disk, and will be streamed only when needed. Delay the time before you need a cluster.

occ-animated

Fast groupby / aggregations

Vaex implements parallelized, highly performant groupby operations, especially when using categories (>1 billion/second).

groupby

Fast and efficient join

Vaex doesn't copy/materialize the 'right' table when joining, saving gigabytes of memory. With subsecond joining on a billion rows, it's pretty fast!

join

More features

Learn how to use Vaex efficiently

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for vaex, version 3.0.0
Filename, size File type Python version Upload date Hashes
Filename, size vaex-3.0.0-py3-none-any.whl (3.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size vaex-3.0.0.tar.gz (3.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page