No project description provided
Project description
Polars
Blazingly fast DataFrames in Rust & Python
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow(2) as memory model.
- Lazy | eager execution
- Multi-threaded
- SIMD
- Query optimization
- Powerful expression API
- Rust | Python | ...
To learn more, read the User Guide.
>>> df = pl.DataFrame(
{
"A": [1, 2, 3, 4, 5],
"fruits": ["banana", "banana", "apple", "apple", "banana"],
"B": [5, 4, 3, 2, 1],
"cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
}
)
# embarrassingly parallel execution
# very expressive query language
>>> (df
.sort("fruits")
.select([
"fruits",
"cars",
lit("fruits").alias("literal_string_fruits"),
col("B").filter(col("cars") == "beetle").sum(),
col("A").filter(col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), # groups by "cars"
col("A").sum().over("fruits").alias("sum_A_by_fruits"), # groups by "fruits"
col("A").reverse().over("fruits").flatten().alias("rev_A_by_fruits"), # groups by "fruits
col("A").sort_by("B").over("fruits").flatten().alias("sort_A_by_B_by_fruits") # groups by "fruits"
]))
shape: (5, 8)
┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────â”
│ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │
│ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │
│ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │
│ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
â•žâ•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•â•â•â•ªâ•â•â•â•â•â•â•â•â•â•â•â•â•â•¡
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │
├╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │
└──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘
Performance 🚀🚀
Polars is very fast, and in fact is one of the best performing solutions available. See the results in h2oai's db-benchmark.
Rust setup
You can take latest release from crates.io
, or if you want to use the latest features/ performance improvements
point to the master
branch of this repo.
polars = {git = "https://github.com/ritchie46/polars", rev = "<optional git tag>" }
Rust version
Required Rust version >=1.52
Python users read this!
Polars is currently transitioning from py-polars
to polars
. Some docs may still refer the old name.
Install the latest polars version with:
$ pip3 install polars
Documentation
Want to know about all the features Polars support? Read the docs!
Rust
Python
- installation guide:
$ pip3 install polars
- User Guide
- Reference guide
Contribution
Want to contribute? Read our contribution guideline.
[Python] compile py-polars from source
If you want a bleeding edge release or maximal performance you should compile py-polars from source.
This can be done by going through the following steps in sequence:
- install the latest rust compiler
$ pip3 install maturin
- Choose any of:
- Very long compile times, fastest binary:
$ cd py-polars && maturin develop --rustc-extra-args="-C target-cpu=native" --release
- Shorter compile times, fast binary:
$ cd py-polars && maturin develop --rustc-extra-args="-C codegen-units=16 -C lto=thin -C target-cpu=native" --release
Note that the Rust crate implementing the Python bindings is called py-polars
to distinguish from the wrapped
Rust crate polars
itself. However, both the Python package and the Python module are named polars
, so you
can pip install polars
and import polars
(previously, these were called py-polars
and pypolars
).
Arrow2
Polars has a fully functional arrow2 branch and will ship the python binaries from this branch. Arrow2 is a faster and safer implementation of the arrow spec. Arrow2 also has a more granular code base, helping to reduce the compiler bloat.
Acknowledgements
Development of Polars is proudly powered by
Sponsors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for polars-0.9.0_beta.1-cp36-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa4e0c43e1fa793bf5ba4838e3e253a1eefe5a7e5453b7796ccce0dde54e9307 |
|
MD5 | 68191f47625c00ba6be3a0e14f4abd4d |
|
BLAKE2b-256 | c9206a0c1d40674c58ebe79e9a769c5834ae1e99c28b4dff530c0065202efda7 |
Hashes for polars-0.9.0_beta.1-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e19ca6b9808423f26851c9a35ca6906192faf236a3371b3f9b18c749af2789c5 |
|
MD5 | 14bdf6110dbc0eae6918f88016687511 |
|
BLAKE2b-256 | debc14cf6cc01be7dc363cf33af08e9ceb650a0d433d3725fbd959b609be2cef |
Hashes for polars-0.9.0_beta.1-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af9a90e9d7a994754dd87946c7a49fc3c4780ac39f5e1dc41838381956029ebc |
|
MD5 | 502b1d716243eebfba0bc5d2aef7ed92 |
|
BLAKE2b-256 | ea674ff38eea9bbd1498f608d5adc35465e065a1759b4af1eba22412f4fd73d2 |