Blazingly fast DataFrame library
Project description
Polars
Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow
Blazingly fast DataFrames in Rust, Python & Node.js
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.
- Lazy | eager execution
- Multi-threaded
- SIMD
- Query optimization
- Powerful expression API
- Rust | Python | ...
To learn more, read the User Guide.
>>> import polars as pl
>>> df = pl.DataFrame(
... {
... "A": [1, 2, 3, 4, 5],
... "fruits": ["banana", "banana", "apple", "apple", "banana"],
... "B": [5, 4, 3, 2, 1],
... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
... }
... )
# embarrassingly parallel execution
# very expressive query language
>>> (
... df
... .sort("fruits")
... .select(
... [
... "fruits",
... "cars",
... pl.lit("fruits").alias("literal_string_fruits"),
... pl.col("B").filter(pl.col("cars") == "beetle").sum(),
... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), # groups by "cars"
... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), # groups by "fruits"
... pl.col("A").reverse().over("fruits").flatten().alias("rev_A_by_fruits"), # groups by "fruits
... pl.col("A").sort_by("B").over("fruits").flatten().alias("sort_A_by_B_by_fruits"), # groups by "fruits"
... ]
... )
... )
shape: (5, 8)
┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐
│ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │
│ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │
│ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │
│ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │
└──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘
Performance 🚀🚀
Polars is very fast, and in fact is one of the best performing solutions available. See the results in h2oai's db-benchmark.
Python setup
Install the latest polars version with:
$ pip3 install polars
Update existing polars installation to the lastest version with:
$ pip3 install -U polars
Releases happen quite often (weekly / every few days) at the moment, so updating polars regularily to get the latest bugfixes / features might not be a bad idea.
Rust setup
You can take latest release from crates.io
, or if you want to use the latest features / performance improvements
point to the master
branch of this repo.
polars = { git = "https://github.com/pola-rs/polars", rev = "<optional git tag>" }
Rust version
Required Rust version >=1.52
Documentation
Want to know about all the features Polars supports? Read the docs!
Python
- Installation guide:
$ pip3 install polars
- Python documentation
- User guide
Rust
Node
- Installation guide:
$ yarn install nodejs-polars
- Node documentation
- User guide
Contribution
Want to contribute? Read our contribution guideline.
[Python]: compile polars from source
If you want a bleeding edge release or maximal performance you should compile polars from source.
This can be done by going through the following steps in sequence:
- Install the latest Rust compiler
- Install maturin:
$ pip3 install maturin
- Choose any of:
- Fastest binary, very long compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C target-cpu=native" --release
- Fast binary, Shorter compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C codegen-units=16 -C lto=thin -C target-cpu=native" --release
- Fastest binary, very long compile times:
Note that the Rust crate implementing the Python bindings is called py-polars
to distinguish from the wrapped
Rust crate polars
itself. However, both the Python package and the Python module are named polars
, so you
can pip install polars
and import polars
.
Arrow2
Polars has transitioned to arrow2. Arrow2 is a faster and safer implementation of the Apache Arrow Columnar Format. Arrow2 also has a more granular code base, helping to reduce the compiler bloat.
Acknowledgements
Development of Polars is proudly powered by
Sponsors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for polars-0.12.9-cp36-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62891d3bf71f0fa28708676403c216fba3f14c9f89f344012c4acd749c8ed589 |
|
MD5 | 2cad0b328423ebc5cbb619d73a6251b6 |
|
BLAKE2b-256 | f077dad870c2961081b131f01cb7dac027b334eb07b12943600b334833992d23 |
Hashes for polars-0.12.9-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0397d7c89bd7b11adc0f30bbdf7870cf07ddfea94b0780690dfd024c7993446 |
|
MD5 | c2cff727b511806bed8c4bcec95d9ea0 |
|
BLAKE2b-256 | 07f3c8cc73e5f2b00525ec748a8814fe2c39cb54333410eca378b9317f735f2c |
Hashes for polars-0.12.9-cp36-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e62c4bd0bcd611691f64fda0108b7b3d072707d867bd6e05b0b8e6c5106d5b6 |
|
MD5 | 47587181bcc911abdaa647ae5a3f5799 |
|
BLAKE2b-256 | d097e0b6ee1ae98a751599ab8a6a72f8f8406935fdbc9cdf2c3b20ca1afda526 |
Hashes for polars-0.12.9-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f10f354943a9422bde0a6aaf6597ab967a859fbc0567671b5be65eea57f6e0a |
|
MD5 | f0e7f844919287c5feb13550523a2049 |
|
BLAKE2b-256 | 8563ac9581056661d7c372fe659449f27b12c54b5f0872aaba4fadf92bfcaa21 |