Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Ibis

Documentation status Project chat Anaconda badge PyPI Build status Build status Codecov branch

What is Ibis?

Ibis is the portable Python dataframe library:

See the documentation on "Why Ibis?" to learn more.

Getting started

You can pip install Ibis with a backend and example data:

pip install 'ibis-framework[duckdb,examples]'

💡 Tip

See the installation guide for more installation options.

Then use Ibis:

>>> import ibis
>>> ibis.options.interactive = True
>>> t = ibis.examples.penguins.fetch()
>>> t
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
 species  island     bill_length_mm  bill_depth_mm  flipper_length_mm  body_mass_g  sex     year  
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
 string   string     float64         float64        int64              int64        string  int64 
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
 Adelie   Torgersen            39.1           18.7                181         3750  male     2007 
 Adelie   Torgersen            39.5           17.4                186         3800  female   2007 
 Adelie   Torgersen            40.3           18.0                195         3250  female   2007 
 Adelie   Torgersen            NULL           NULL               NULL         NULL  NULL     2007 
 Adelie   Torgersen            36.7           19.3                193         3450  female   2007 
 Adelie   Torgersen            39.3           20.6                190         3650  male     2007 
 Adelie   Torgersen            38.9           17.8                181         3625  female   2007 
 Adelie   Torgersen            39.2           19.6                195         4675  male     2007 
 Adelie   Torgersen            34.1           18.1                193         3475  NULL     2007 
 Adelie   Torgersen            42.0           20.2                190         4250  NULL     2007 
                                                                                          
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘
>>> g = t.group_by("species", "island").agg(count=t.count()).order_by("count")
>>> g
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
 species    island     count 
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
 string     string     int64 
├───────────┼───────────┼───────┤
 Adelie     Biscoe        44 
 Adelie     Torgersen     52 
 Adelie     Dream         56 
 Chinstrap  Dream         68 
 Gentoo     Biscoe       124 
└───────────┴───────────┴───────┘

💡 Tip

See the getting started tutorial for a full introduction to Ibis.

Python + SQL: better together

For most backends, Ibis works by compiling its dataframe expressions into SQL:

>>> ibis.to_sql(g)
SELECT
  "t1"."species",
  "t1"."island",
  "t1"."count"
FROM (
  SELECT
    "t0"."species",
    "t0"."island",
    COUNT(*) AS "count"
  FROM "penguins" AS "t0"
  GROUP BY
    1,
    2
) AS "t1"
ORDER BY
  "t1"."count" ASC

You can mix SQL and Python code:

>>> a = t.sql("SELECT species, island, count(*) AS count FROM penguins GROUP BY 1, 2")
>>> a
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
 species    island     count 
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
 string     string     int64 
├───────────┼───────────┼───────┤
 Adelie     Torgersen     52 
 Adelie     Biscoe        44 
 Adelie     Dream         56 
 Gentoo     Biscoe       124 
 Chinstrap  Dream         68 
└───────────┴───────────┴───────┘
>>> b = a.order_by("count")
>>> b
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
 species    island     count 
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
 string     string     int64 
├───────────┼───────────┼───────┤
 Adelie     Biscoe        44 
 Adelie     Torgersen     52 
 Adelie     Dream         56 
 Chinstrap  Dream         68 
 Gentoo     Biscoe       124 
└───────────┴───────────┴───────┘

This allows you to combine the flexibility of Python with the scale and performance of modern SQL.

Backends

Ibis supports 20 backends:

How it works

Most Python dataframes are tightly coupled to their execution engine. And many databases only support SQL, with no Python API. Ibis solves this problem by providing a common API for data manipulation in Python, and compiling that API into the backend’s native language. This means you can learn a single API and use it across any supported backend (execution engine).

Ibis broadly supports two types of backend:

  1. SQL-generating backends
  2. DataFrame-generating backends

Ibis backend types

Portability

To use different backends, you can set the backend Ibis uses:

>>> ibis.set_backend("duckdb")
>>> ibis.set_backend("polars")
>>> ibis.set_backend("datafusion")

Typically, you'll create a connection object:

>>> con = ibis.duckdb.connect()
>>> con = ibis.polars.connect()
>>> con = ibis.datafusion.connect()

And work with tables in that backend:

>>> con.list_tables()
['penguins']
>>> t = con.table("penguins")

You can also read from common file formats like CSV or Apache Parquet:

>>> t = con.read_csv("penguins.csv")
>>> t = con.read_parquet("penguins.parquet")

This allows you to iterate locally and deploy remotely by changing a single line of code.

💡 Tip

Check out the blog on backend agnostic arrays for one example using the same code across DuckDB and BigQuery.

Community and contributing

Ibis is an open source project and welcomes contributions from anyone in the community.

Join our community by interacting on GitHub or chatting with us on Zulip.

For more information visit https://ibis-project.org/.

Governance

The Ibis project is an independently governed open source community project to build and maintain the portable Python dataframe library. Ibis has contributors across a range of data companies and institutions.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ibis_framework-11.0.1.dev133.tar.gz (1.3 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

ibis_framework-11.0.1.dev133-py3-none-any.whl (2.0 MB view details)

Uploaded Python 3

File details

Details for the file ibis_framework-11.0.1.dev133.tar.gz.

File metadata

  • Download URL: ibis_framework-11.0.1.dev133.tar.gz
  • Upload date:
  • Size: 1.3 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.11

File hashes

Hashes for ibis_framework-11.0.1.dev133.tar.gz
Algorithm Hash digest
SHA256 f1db83c9ca9eba49c61a5ab76ec8ee92c157f460b37ee5d958f406c6304fe01c
MD5 d99653c2d215a7e24d266484dcda9db3
BLAKE2b-256 b378e35cd19e28b7ab3b89a5a00d0e7cd3a48872c5effd0d649efa4cf511a017

See more details on using hashes here.

Provenance

The following attestation bundles were made for ibis_framework-11.0.1.dev133.tar.gz:

Publisher: pre-release.yml on ibis-project/ibis

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file ibis_framework-11.0.1.dev133-py3-none-any.whl.

File metadata

File hashes

Hashes for ibis_framework-11.0.1.dev133-py3-none-any.whl
Algorithm Hash digest
SHA256 c52c761895cf25a343308c213583ff681118048732a8ca7b67db8e6079d24d69
MD5 9f7009846d269f02bb44d4345bf45a52
BLAKE2b-256 82bd171864bf85653923d88d183b6670059451daabf91ca31850ef216b9cfa85

See more details on using hashes here.

Provenance

The following attestation bundles were made for ibis_framework-11.0.1.dev133-py3-none-any.whl:

Publisher: pre-release.yml on ibis-project/ibis

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

12.0.0

2 files

This release

11.0.1.dev133 This release

2 files

11.0.0

2 files

10.8.0

2 files

10.7.0

2 files

10.6.0

2 files

10.5.0

2 files

10.4.0

2 files

10.3.1

2 files

10.3.0

2 files

10.2.0

2 files

10.1.0

2 files

10.0.0

2 files

9.5.0

2 files

9.4.0

2 files

9.3.0

2 files

9.2.0

2 files

9.1.0

2 files

9.0.0

2 files

8.0.0

2 files

7.2.0

2 files

7.1.0

2 files

7.0.0

2 files

6.2.0

2 files

6.1.0

2 files

6.0.0

2 files

5.1.0

2 files

5.0.0

2 files

4.1.0

2 files

4.0.0

2 files

3.2.0

2 files

3.1.0

2 files

3.0.2

2 files

3.0.1

2 files

3.0.0

2 files

2.1.1

2 files

2.1.0

2 files

2.0.0

2 files

1.4.0

2 files

1.3.0

2 files

1.2.0

2 files

1.1.0

2 files

1.0.0

2 files

0.14.0

2 files

0.13.0

1 file

0.12.0

1 file

0.11.2

1 file

0.11.1

1 file

0.10.0

1 file

0.9.0

1 file

0.8.1

1 file

0.8.0

1 file

0.7.1

1 file

0.7.0

1 file

0.6.1

1 file

v0.6.0

1 file

0.5.2

1 file

0.5.1

1 file

0.4.1

1 file

0.4.0

1 file

0.3.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page