Blazingly fast DataFrame library
Project description
Polars
Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow
Blazingly fast DataFrames in Rust, Python & Node.js
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.
- Lazy | eager execution
- Multi-threaded
- SIMD
- Query optimization
- Powerful expression API
- Rust | Python | ...
To learn more, read the User Guide.
>>> import polars as pl
>>> df = pl.DataFrame(
... {
... "A": [1, 2, 3, 4, 5],
... "fruits": ["banana", "banana", "apple", "apple", "banana"],
... "B": [5, 4, 3, 2, 1],
... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
... }
... )
# embarrassingly parallel execution
# very expressive query language
>>> (
... df
... .sort("fruits")
... .select(
... [
... "fruits",
... "cars",
... pl.lit("fruits").alias("literal_string_fruits"),
... pl.col("B").filter(pl.col("cars") == "beetle").sum(),
... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), # groups by "cars"
... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), # groups by "fruits"
... pl.col("A").reverse().over("fruits").flatten().alias("rev_A_by_fruits"), # groups by "fruits
... pl.col("A").sort_by("B").over("fruits").flatten().alias("sort_A_by_B_by_fruits"), # groups by "fruits"
... ]
... )
... )
shape: (5, 8)
┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐
│ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │
│ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │
│ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │
│ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │
└──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘
Performance 🚀🚀
Polars is very fast, and in fact is one of the best performing solutions available. See the results in h2oai's db-benchmark.
Python setup
Install the latest polars version with:
$ pip3 install polars
Update existing polars installation to the lastest version with:
$ pip3 install -U polars
Releases happen quite often (weekly / every few days) at the moment, so updating polars regularily to get the latest bugfixes / features might not be a bad idea.
Rust setup
You can take latest release from crates.io
, or if you want to use the latest features / performance improvements
point to the master
branch of this repo.
polars = { git = "https://github.com/pola-rs/polars", rev = "<optional git tag>" }
Rust version
Required Rust version >=1.58
Documentation
Want to know about all the features Polars supports? Read the docs!
Python
- Installation guide:
$ pip3 install polars
- Python documentation
- User guide
Rust
Node
- Installation guide:
$ yarn install nodejs-polars
- Node documentation
- User guide
Contribution
Want to contribute? Read our contribution guideline.
[Python]: compile polars from source
If you want a bleeding edge release or maximal performance you should compile polars from source.
This can be done by going through the following steps in sequence:
- Install the latest Rust compiler
- Install maturin:
$ pip3 install maturin
- Choose any of:
- Fastest binary, very long compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C target-cpu=native" --release
- Fast binary, Shorter compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C codegen-units=16 -C lto=thin -C target-cpu=native" --release
- Fastest binary, very long compile times:
Note that the Rust crate implementing the Python bindings is called py-polars
to distinguish from the wrapped
Rust crate polars
itself. However, both the Python package and the Python module are named polars
, so you
can pip install polars
and import polars
.
Arrow2
Polars has transitioned to arrow2. Arrow2 is a faster and safer implementation of the Apache Arrow Columnar Format. Arrow2 also has a more granular code base, helping to reduce the compiler bloat.
Acknowledgements
Development of Polars is proudly powered by
Sponsors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for polars-0.13.5-cp36-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e902ad2d4aa33e2e298700b7064aef1787630e38c658a2c559b8c18eb2216e6 |
|
MD5 | 65ad5ba042a709d56e7bc33239091c56 |
|
BLAKE2b-256 | d6929b16a3d85017569c8b661e46f5851c154f1e884d5ef8f9cafd145596044c |
Hashes for polars-0.13.5-cp36-abi3-manylinux_2_24_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c049818da94ba11da26613c62ba9995025d4e8a2a4834d13084868d20cf606aa |
|
MD5 | 34413c82dad77d6be222972d30c3a0b1 |
|
BLAKE2b-256 | eb0a5e21ce551dbac9ab0a6c63a02748c7631bd1bacdfff70b545fc9c0783c2a |
Hashes for polars-0.13.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07a82e7e465ad4490a94f53f27d01f4b0a14142d77c00a4b83a29dca44cb80f6 |
|
MD5 | 59349daa42069a207e95d042b2f02497 |
|
BLAKE2b-256 | ddd63206a093f77ccdc2e38ba9315c6907c55e3744fce11d54bf91586558405c |
Hashes for polars-0.13.5-cp36-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7eb9719cc0b59a66296eb4f89a49b8f3d0229866c86713edadc0a159999c7948 |
|
MD5 | 1eb13fdebf84f0933f85333d6e97d1b0 |
|
BLAKE2b-256 | a09d9cf918b0c8a91bdd4d681befd5cd56ef11b69efd9b22cb13ee874c5736a7 |
Hashes for polars-0.13.5-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67626594b078a9340eb588f88770901d94ff29a538bdc865c761e0c3e780e73e |
|
MD5 | 74d67ca9a6db4693a220bdef13506802 |
|
BLAKE2b-256 | 75e7960f7cc1459da95c6b6198088d8efaf871571d3d2c5109b03577f73c385f |