Blazingly fast DataFrame library
Project description
Polars
Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow
Blazingly fast DataFrames in Rust & Python
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.
- Lazy | eager execution
- Multi-threaded
- SIMD
- Query optimization
- Powerful expression API
- Rust | Python | ...
To learn more, read the User Guide.
>>> import polars as pl
>>> df = pl.DataFrame(
... {
... "A": [1, 2, 3, 4, 5],
... "fruits": ["banana", "banana", "apple", "apple", "banana"],
... "B": [5, 4, 3, 2, 1],
... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
... }
... )
# embarrassingly parallel execution
# very expressive query language
>>> (
... df
... .sort("fruits")
... .select(
... [
... "fruits",
... "cars",
... pl.lit("fruits").alias("literal_string_fruits"),
... pl.col("B").filter(pl.col("cars") == "beetle").sum(),
... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), # groups by "cars"
... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), # groups by "fruits"
... pl.col("A").reverse().over("fruits").flatten().alias("rev_A_by_fruits"), # groups by "fruits
... pl.col("A").sort_by("B").over("fruits").flatten().alias("sort_A_by_B_by_fruits"), # groups by "fruits"
... ]
... )
... )
shape: (5, 8)
┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐
│ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │
│ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │
│ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │
│ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │
└──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘
Performance 🚀🚀
Polars is very fast, and in fact is one of the best performing solutions available. See the results in h2oai's db-benchmark.
Python setup
Install the latest polars version with:
$ pip3 install polars
Update existing polars installation to the lastest version with:
$ pip3 install -U polars
Releases happen quite often (weekly / every few days) at the moment, so updating polars regularily to get the latest bugfixes / features might not be a bad idea.
Rust setup
You can take latest release from crates.io
, or if you want to use the latest features / performance improvements
point to the master
branch of this repo.
polars = { git = "https://github.com/pola-rs/polars", rev = "<optional git tag>" }
Rust version
Required Rust version >=1.52
Documentation
Want to know about all the features Polars supports? Read the docs!
Python
- Installation guide:
$ pip3 install polars
- Python documentation
- User guide
Rust
Contribution
Want to contribute? Read our contribution guideline.
[Python]: compile polars from source
If you want a bleeding edge release or maximal performance you should compile polars from source.
This can be done by going through the following steps in sequence:
- Install the latest Rust compiler
- Install maturin:
$ pip3 install maturin
- Choose any of:
- Fastest binary, very long compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C target-cpu=native" --release
- Fast binary, Shorter compile times:
$ cd py-polars && maturin develop --rustc-extra-args="-C codegen-units=16 -C lto=thin -C target-cpu=native" --release
- Fastest binary, very long compile times:
Note that the Rust crate implementing the Python bindings is called py-polars
to distinguish from the wrapped
Rust crate polars
itself. However, both the Python package and the Python module are named polars
, so you
can pip install polars
and import polars
.
Arrow2
Polars has transitioned to arrow2. Arrow2 is a faster and safer implementation of the Apache Arrow Columnar Format. Arrow2 also has a more granular code base, helping to reduce the compiler bloat.
Acknowledgements
Development of Polars is proudly powered by
Sponsors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for polars-0.12.2-cp36-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18a41900fd058eec9ab24ef58a6eb772a6485bf6e6abd2284f97890e1bccc581 |
|
MD5 | 95a3b346e005d7ee407fbeb1b15fc392 |
|
BLAKE2b-256 | ec8fb18a69f326947f52a856f7133e4b1219464a078bacebf121f8ceae6234ae |
Hashes for polars-0.12.2-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18a5d329a19dfe3875bce2fab4b9c8c543142aed4dab638dac899f9560081dac |
|
MD5 | 349da7ebe329a57c77f4419641d25ec9 |
|
BLAKE2b-256 | 531638b797c68ff594cd29f51ae061f13fa633da9ff82f40fa9d60bd17087f05 |
Hashes for polars-0.12.2-cp36-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 723ca0b1cc77cd4eaef13795d3506cde7d462d3daf85f5f0aa902731a79b081d |
|
MD5 | 0b258d999e8d3e931145bd3ff4fe2833 |
|
BLAKE2b-256 | 58f17828b740a58833058f481d1ae197e17ea23ca0527c0e5f65196e229fe0be |
Hashes for polars-0.12.2-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddc07b12debac0fef6a05e026a32ff733d5c91b9b650f38b584983b8b8e84eb9 |
|
MD5 | 3475733ecdacbeb642bdf567a9df2832 |
|
BLAKE2b-256 | f681d6ca94eb1aa7fa66c179e29107e449ab163523a64abb36879001ee9de343 |