tidy3

dplyr for Python, on Polars. Write R-style pipes in a notebook, one verb per line with comments in between, see every column's type, and hand the result to plot3, scikit-learn, NumPy, or pandas without leaving the pipe.

pip install tidy3

Contents: In a notebook · From R · Lazy and fast · Reshape · Plot · scikit-learn · Scripts · Full reference

In a notebook

Load the extension once, in a cell of its own:

%load_ext tidy3.jupyter
from tidy3 import *

From the next cell on, a pipe runs over as many lines as you like, each line starting with >>, comments and blank lines included, and column names go bare, as in R. (The cell that loads the extension can't use them yet: IPython reads a whole cell before running any of it.)

penguins = scan_csv(
    "https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/inst/extdata/penguins.csv",
    null_values="NA",
) >> compute()

# Heaviest groups first
penguins
>> drop_na(sex)                        # 11 penguins have no recorded sex
>> group_by(species, sex)
>> summarise(
    n = n(),
    bill_mm = mean(bill_length_mm).round(1),
    mass_kg = (mean(body_mass_g) / 1000).round(2),
)
>> arrange(desc(mass_kg))

TidyFrame (preview groups=['species'])
shape: (6, 5)
┌───────────┬────────┬─────┬─────────┬─────────┐
│ species   ┆ sex    ┆ n   ┆ bill_mm ┆ mass_kg │
│ ---       ┆ ---    ┆ --- ┆ ---     ┆ ---     │
│ str       ┆ str    ┆ u32 ┆ f64     ┆ f64     │
╞═══════════╪════════╪═════╪═════════╪═════════╡
│ Gentoo    ┆ male   ┆ 61  ┆ 49.5    ┆ 5.48    │
│ Gentoo    ┆ female ┆ 58  ┆ 45.6    ┆ 4.68    │
│ Adelie    ┆ male   ┆ 73  ┆ 40.4    ┆ 4.04    │
│ Chinstrap ┆ male   ┆ 34  ┆ 51.1    ┆ 3.94    │
│ Chinstrap ┆ female ┆ 34  ┆ 46.6    ┆ 3.53    │
│ Adelie    ┆ female ┆ 73  ┆ 37.3    ┆ 3.37    │
└───────────┴────────┴─────┴─────────┴─────────┘

Every table shows each column's type under its name, and the caption keeps the grouping that's left (summarise drops the last group, as in dplyr). glimpse(penguins) prints one line per column. To check part of a long pipe, run its first lines with %%tidy3_run.

If you know dplyr

The verbs, helpers, and arguments are dplyr's and tidyr's, with >> for R's |>:

As in dplyr, a column beats a notebook variable of the same name; write env.x for the variable. Python's own spellings differ in a few places: na_rm for na.rm, True for TRUE, and _x/_y join suffixes for .x/.y.

Lazy until you need rows

A pipe is a plan, not a computation. Polars optimises the whole plan, reads only the columns it uses, and pushes filters into the file scan. Rows are computed when something needs them:

• a preview: a notebook table shows its first rows, and its shape counts all of them, as a tibble does (… 20 more rows). On a very large query, options(preview_count=False) skips the count and shows ?,
• a plot, to_numpy(), or write_parquet() / write_csv(),
• collect(), which returns a Polars, pandas, Arrow, or NumPy result,
• compute(), which runs the plan so far and keeps piping from the in-memory result. Use it for a table you'll reuse, like penguins above: the CSV downloads once, not once per step.

penguins.collect()                  # Polars DataFrame
penguins.collect(as_="pandas")      # pandas DataFrame

On 10 million rows, a filter → mutate → group_by → summarise → arrange pipeline takes 41 ms in tidy3, the same as hand-written Polars (42 ms), against 60 ms in pandas and 166 ms in datar (benchmarks).

Reshape for plotting

pivot_longer turns the three measurement columns into one, the shape facets want:

long = penguins
>> mutate(id = row_number())
>> select(id, species, ends_with("_mm"))
>> pivot_longer(ends_with("_mm"), names_to = "measure", values_to = "mm",
                values_drop_na = True)

long >> head(4)

┌─────┬─────────┬───────────────────┬───────┐
│ id  ┆ species ┆ measure           ┆ mm    │
│ --- ┆ ---     ┆ ---               ┆ ---   │
│ i64 ┆ str     ┆ str               ┆ f64   │
╞═════╪═════════╪═══════════════════╪═══════╡
│ 1   ┆ Adelie  ┆ bill_length_mm    ┆ 39.1  │
│ 1   ┆ Adelie  ┆ bill_depth_mm     ┆ 18.7  │
│ 1   ┆ Adelie  ┆ flipper_length_mm ┆ 181.0 │
│ 2   ┆ Adelie  ┆ bill_length_mm    ┆ 39.5  │
└─────┴─────────┴───────────────────┴───────┘

A pipe flows into plot3's ggplot(), and a line starting with + adds a layer, as in R:

from plot3 import *

long
>> ggplot(aes(x = species, y = mm, fill = species))
+ geom_boxplot()
+ facet_wrap("measure", scales = "free_y")   # one panel per measurement
+ scale_fill_hue()                           # ggplot2's default colours
+ theme_grey()                               # and its grey panel
+ theme(legend_position = "none")

plot3 has ggplot2's themes and scales, so a figure can look exactly like one from R. The other figures here use theme_dark(), the look of plot3's interactive viewer.

pivot_wider, separate, unite, nest/unnest, complete, and the rest of tidyr are in the reference.

Plot with plot3

One scatter shows Simpson's paradox: over all penguins, longer bills are shallower (grey), but within each species they are deeper:

penguins
>> drop_na(bill_length_mm, bill_depth_mm)
>> ggplot(aes(x = bill_length_mm, y = bill_depth_mm, colour = species))
+ geom_point(alpha = 0.6)
+ geom_smooth(method = "lm")                     # one trend per species
+ geom_smooth(method = "lm", colour = "grey70")  # and one for all penguins

Figures are interactive (zoom, hover, orbit in 3D) and save for papers with ggsave("fig.pdf", p). plot3's animations work the same way, here gapminder's life expectancy against income, year by year:

gapminder = scan_csv("https://raw.githubusercontent.com/kirenz/datasets/master/gapminder.csv")

gapminder
>> mutate(pop_millions = pop / 1e6)
>> ggplot(aes(x = gdpPercap, y = lifeExp, size = pop_millions,
              colour = continent, group = country))
+ geom_point(alpha = 0.7)
+ scale_x_log10()
+ transition_time("year")
+ labs(title = "{frame_time}", x = "GDP per capita", y = "Life expectancy")

Hand off to scikit-learn, NumPy, and pandas

tidy3 does the cleaning and scaling, to_numpy() hands scikit-learn a plain matrix, and the model's output comes straight back as a column. scale() is R's z-score and rescale(to=(0, 1)) is min-max scaling; both skip missing values and work by group. Here k-means finds three groups in the penguins' bill and flipper sizes:

from sklearn.cluster import KMeans

features = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm"]
complete = penguins >> drop_na(all_of(features)) >> compute()

X = complete
>> mutate(across(all_of(features), scale))
>> to_numpy(columns = features)

kmeans = KMeans(n_clusters=3, n_init=10, random_state=1).fit(X)

clustered = complete >> mutate(cluster = kmeans.labels_ + 1)   # labels come back
clustered >> count(species, cluster)

┌───────────┬─────────┬─────┐
│ species   ┆ cluster ┆ n   │
│ ---       ┆ ---     ┆ --- │
│ str       ┆ i32     ┆ u32 │
╞═══════════╪═════════╪═════╡
│ Adelie    ┆ 1       ┆ 146 │
│ Adelie    ┆ 3       ┆ 5   │
│ Chinstrap ┆ 1       ┆ 5   │
│ Chinstrap ┆ 3       ┆ 63  │
│ Gentoo    ┆ 2       ┆ 123 │
└───────────┴─────────┴─────┘

Note: Use custom normalizations as follows

z_score = lambda x: (x - mean(x)) / sd(x)
complete >> mutate(across(all_of(features), z_score))

The clusters match the species for 332 of 342 penguins. The centres, in millimetres, are a group_by, not an inverse transform:

centres = clustered >> group_by(cluster) >> summarise(across(all_of(features), mean))

clustered
>> ggplot(aes(x = flipper_length_mm, y = bill_length_mm, colour = factor(cluster)))
+ geom_point(aes(shape = species), alpha = 0.75)
+ geom_point(data = centres, size = 16, shape = "cross", colour = "grey70")

The whole example, as a script, is examples/cluster_penguins.py (pip install scikit-learn). Other handoffs:

In a Python script

A plain .py file is ordinary Python: wrap a multi-line pipe in parentheses and name columns with col("…") in expressions and strings elsewhere. This form works everywhere, notebooks included, and mixes with bare names:

result = (
    penguins
    >> filter(col("species") == "Gentoo")
    >> group_by("island")
    >> summarise(heaviest=max("body_mass_g"))
)

Or keep the notebook style and run the script with python -m tidy3 run job.py, which applies the same rewriting. nb_export turns a notebook into a plain script, and VS Code setup, %%tidy3_run, and the bare-name rules are in docs/notebooks.md. Running on a remote GPU kernel under CRAFT or SolveIt: docs/craft.md.

More

• Reference: every verb family, the API table, pandas input and output, joins, across, row-wise work, and performance controls.
• Benchmarks and R parity: how tidy3 is timed, and the suite that checks its results against dplyr and tidyr.
• Changelog

Development

git clone https://github.com/rleyvasal/tidy3 && cd tidy3
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python -m pytest -q            # includes every example in this README and docs/

Releases go to PyPI from a version tag; see docs/releasing.md.

License

MIT. See LICENSE.