Narwhals

narwhals_small

PyPI version Downloads Trusted publishing PYPI - Types LFX Health Score OpenSSF Scorecard

Extremely lightweight and extensible compatibility layer between dataframe libraries!

  • Full API support: cuDF, Modin, pandas, Polars, PyArrow.
  • Lazy-only support: Daft, Dask, DuckDB, Ibis, PySpark, SQLFrame.

Seamlessly support all, without depending on any!

  • ✅ Just use a subset of the Polars API, no need to learn anything new
  • ✅ Zero dependencies, Narwhals only uses what the user passes in so your library can stay lightweight
  • ✅ Separate lazy and eager APIs, use expressions
  • ✅ Support pandas' complicated type system and index, without either getting in the way
  • ✅ 100% branch coverage, tested against pandas and Polars nightly builds
  • ✅ Negligible overhead, see overhead
  • ✅ Let your IDE help you thanks to full static typing, see typing
  • ✅ Perfect backwards compatibility policy, see stable api for how to opt-in

Get started!

Table of contents

Installation

  • pip (recommended, as it's the most up-to-date) 
    pip install narwhals
  • conda-forge (also fine, but the latest version may take longer to appear) 
    conda install -c conda-forge narwhals

Usage

There are three steps to writing dataframe-agnostic code using Narwhals:

  1. use narwhals.from_native to wrap a pandas/Polars/Modin/cuDF/PyArrow DataFrame/LazyFrame in a Narwhals class

  2. use the subset of the Polars API supported by Narwhals

  3. use narwhals.to_native to return an object to the user in its original dataframe flavour. For example:

    • if you started with pandas, you'll get pandas back
    • if you started with Polars, you'll get Polars back
    • if you started with Modin, you'll get Modin back (and compute will be distributed)
    • if you started with cuDF, you'll get cuDF back (and compute will happen on GPU)
    • if you started with PyArrow, you'll get PyArrow back

narwhals_gif

Example

Narwhals allows you to define dataframe-agnostic functions. For example:

import narwhals as nw
from narwhals.typing import IntoFrameT


def agnostic_function(df_native: IntoFrameT) -> IntoFrameT:
    return (
        nw.from_native(df_native)
        .with_columns(
            category=nw.when(nw.col("animal").str.contains("whale"))
            .then(nw.lit("whale"))
            .otherwise(nw.lit("other"))
        )
        .to_native()
    )

You can then pass pandas.DataFrame, polars.DataFrame, polars.LazyFrame, duckdb.DuckDBPyRelation, pyspark.sql.DataFrame, pyarrow.Table, and more, to agnostic_function. In each case, no additional dependencies will be required, and computation will stay native to the input library:

import duckdb
import polars as pl
import pandas as pd

data = {
    "animal": ["blue whale", "orca", "dolphin", "humpback whale", "seal"],
    "length_m": [30.0, 8, 2.5, 17, 2.2],
    "weight_kg": [150000, 4000, 200, 30000, 85],
}

print("Polars result")
df_pl = pl.DataFrame(data)
print(agnostic_function(df_pl))

print("DuckDB result")
print(agnostic_function(duckdb.sql("select * from df_pl")))

print("pandas result")
df_pd = pd.DataFrame(data)
print(agnostic_function(df_pd))

Polars result
shape: (5, 4)
┌────────────────┬──────────┬───────────┬──────────┐
│ animal         ┆ length_m ┆ weight_kg ┆ category │
│ ---            ┆ ---      ┆ ---       ┆ ---      │
│ str            ┆ f64      ┆ i64       ┆ str      │
╞════════════════╪══════════╪═══════════╪══════════╡
│ blue whale     ┆ 30.0     ┆ 150000    ┆ whale    │
│ orca           ┆ 8.0      ┆ 4000      ┆ other    │
│ dolphin        ┆ 2.5      ┆ 200       ┆ other    │
│ humpback whale ┆ 17.0     ┆ 30000     ┆ whale    │
│ seal           ┆ 2.2      ┆ 85        ┆ other    │
└────────────────┴──────────┴───────────┴──────────┘
DuckDB result
┌────────────────┬──────────┬───────────┬──────────┐
│     animal     │ length_m │ weight_kg │ category │
│    varchar     │  double  │   int64   │ varchar  │
├────────────────┼──────────┼───────────┼──────────┤
│ blue whale     │     30.0 │    150000 │ whale    │
│ orca           │      8.0 │      4000 │ other    │
│ dolphin        │      2.5 │       200 │ other    │
│ humpback whale │     17.0 │     30000 │ whale    │
│ seal           │      2.2 │        85 │ other    │
└────────────────┴──────────┴───────────┴──────────┘

pandas result
           animal  length_m  weight_kg category
0      blue whale      30.0     150000    whale
1            orca       8.0       4000    other
2         dolphin       2.5        200    other
3  humpback whale      17.0      30000    whale
4            seal       2.2         85    other

See the tutorial for several examples!

Scope

  • Do you maintain a dataframe-consuming library?
  • Do you have a specific Polars function in mind that you would like Narwhals to have in order to make your work easier?

If you said yes to both, we'd love to hear from you!

Roadmap

See roadmap discussion on GitHub for an up-to-date plan of future work.

Used by

Join the party!

Feel free to add your project to the list if it's missing, and/or chat with us on Discord if you'd like any support.

Sponsors and institutional partners

Narwhals is 100% independent, community-driven, and community-owned. We are extremely grateful to the following organisations for having provided some funding / development time:

If you contribute to Narwhals on your organization's time, please let us know. We'd be happy to add your employer to this list!

Support

If you'd like to say "thank you", please give us a ⭐ star ⭐.

Please contact hello_narwhals@proton.me if you would like to:

  • Receive professional support (e.g., if you're using or would like to use Narwhals at your company).
  • Have any Narwhals fixes / features prioritised.
  • Commission any Narwhals plugins for new backends.

Appears on

Narwhals has been featured in several talks, podcasts, and blog posts:

Why "Narwhals"?

Coz they are so awesome.

Thanks to Olha Urdeichuk for the illustration!

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

3 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for EdoAbati from gravatar.com EdoAbati Avatar for FBruzzesi from gravatar.com FBruzzesi Avatar for MarcoGorelli from gravatar.com MarcoGorelli

Credits

Author: Marco Gorelli

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

cudf dask duckdb ibis modin pandas polars pyarrow pyspark pyspark-connect sql sqlframe

Tags

dataframes interoperability pandas polars pyarrow dask modin cudf

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

narwhals-2.24.0.tar.gz (661.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

narwhals-2.24.0-py3-none-any.whl (461.0 kB view details)

Uploaded Python 3

File details

Details for the file narwhals-2.24.0.tar.gz.

File metadata

  • Download URL: narwhals-2.24.0.tar.gz
  • Upload date:
  • Size: 661.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for narwhals-2.24.0.tar.gz
Algorithm Hash digest
SHA256 b5c0f684ccd9d7475b564111e319a4964abcf2baf79d3cf6b1003d06ac9b828d
MD5 634590f312da0de93676cbb43f2997a8
BLAKE2b-256 2b1d58946e5aab18393e793bd4add6985b95d0e01c3a2d832f38f54468b10dcd

See more details on using hashes here.

Provenance

The following attestation bundles were made for narwhals-2.24.0.tar.gz:

Publisher: publish_to_pypi.yml on narwhals-dev/narwhals

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file narwhals-2.24.0-py3-none-any.whl.

File metadata

  • Download URL: narwhals-2.24.0-py3-none-any.whl
  • Upload date:
  • Size: 461.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for narwhals-2.24.0-py3-none-any.whl
Algorithm Hash digest
SHA256 42fdedf44e5b2ca7505630d45b4ac3058f38d8485cba9fe1652ca23152df7489
MD5 ed7bc335c82830984240163f7885c4d0
BLAKE2b-256 7e85a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e

See more details on using hashes here.

Provenance

The following attestation bundles were made for narwhals-2.24.0-py3-none-any.whl:

Publisher: publish_to_pypi.yml on narwhals-dev/narwhals

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

2.24.0

2.23.0

2.22.1

2.22.0

2.21.2

2.21.0

2.20.0

2.19.0

2.18.1

2.18.0

2.17.0

2.16.0

2.15.0

2.14.0

2.13.0

2.12.0

2.11.0

2.10.2

2.10.1

2.10.0

2.9.0

2.8.0

2.7.0

2.6.0

2.5.0

2.4.0

2.3.0

2.2.0

2.1.2

2.1.1

2.1.0

2.0.1

2.0.0

1.48.1

1.48.0

1.47.1

1.47.0

1.46.0

1.45.0

1.44.0

1.43.1

1.43.0 yanked

Reason this release was yanked:

Unexpected CI failure for Bokeh

1.42.1

1.42.0

1.41.1

1.41.0

1.40.0

1.39.1

1.39.0

1.38.2

1.38.1

1.38.0

1.37.1

1.37.0

1.36.0

1.35.0

1.34.1

1.34.0

1.33.0

1.32.0

1.31.0

1.30.0

1.29.1

1.29.0

1.28.0

1.27.1

1.27.0 yanked

Reason this release was yanked:

bug

1.26.0

1.25.2

1.25.1

1.25.0

1.24.2

1.24.1

1.24.0

1.23.0

1.22.0

1.21.1

1.21.0

1.20.1

1.20.0

1.19.1

1.19.0

1.18.4

1.18.3

1.18.2

1.18.1 yanked

Reason this release was yanked:

Sigstore failed in GitHub - we'll publish 1.18.2 and make sure it's signed

1.18.0 yanked

Reason this release was yanked:

Mistake in the release process, so this release didn't get signed properly on GitHub. We'll release 1.18.1 to remedy

1.17.0

1.16.0

1.15.2

1.15.1

1.15.0

1.14.3

1.14.2

1.14.1

1.14.0

1.13.5

1.13.4

1.13.3

1.13.2

1.13.1

1.12.1

1.12.0

1.11.1

1.11.0

1.10.0

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.0

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.2

1.4.1

1.4.0

1.3.0

1.2.0

1.1.9

1.1.8

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

0.9.29

0.9.28

0.9.27

0.9.26

0.9.25

0.9.24

0.9.23

0.9.22

0.9.21

0.9.20

0.9.19

0.9.18

0.9.17

0.9.16

0.9.15

0.9.14

0.9.13

0.9.12

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.2

0.9.1

0.9.0

0.8.21

0.8.20

0.8.19

0.8.18

0.8.17

0.8.16

0.8.15

0.8.14

0.8.13

0.8.12

0.8.11

0.8.10

0.8.9

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.18

0.7.16

0.7.15

0.7.14

0.7.12

0.7.11

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.1

0.4.0

0.3.0

0.2.0

0.1.14

0.1.13

0.1.12

0.1.11

0.1.10

0.1.9

0.1.8

0.1.7

0.1.6

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page