TurboFastApply

A drop-in, accelerated replacement for pandas.apply().

Status: feature-complete (all 8 roadmap phases) — numeric transforms detected as linear/abs (x * 2 + 1) and row-wise DataFrame functions that are linear combinations of columns (row['a'] + row['b']) run through genuine native fast paths, ~6.2x and ~4.6x faster than plain pandas.apply() on 1,000 rows respectively (examples/benchmark.py). A dedicated small-data optimization pass (dedup'd array/Index construction, replacing numpy scalar calls with plain Python arithmetic in the hot verification path, avoiding redundant column-selection/NaN scans) also closed a real regression that used to exist from 50 rows onward — see "Benchmarks: previous vs now" below for exact numbers. A follow-up pass replaced the row-wise path's real per-row pandas Series construction (for its correctness-required probe/verification rows) with a lightweight dict-based stand-in (_RowView in decide_row.py), cutting that fixed cost by 3-4x and lowering its engine="auto" threshold (MIN_ROWS) from an interim 300 back down to 100 — now a genuine win from 50 rows up, not just parity. See "Benchmarks: previous vs now" for the full before/after. Arbitrary callables that don't qualify get a sampling-based threaded fallback that only engages when actually measured faster (helps I/O-bound work, correctly declines pure-CPU-bound Python). Everything else falls back to native pandas.apply(), so .turbofastapply is always correctness-equivalent to it. See claude.md for the full roadmap, including three real correctness bugs found and fixed during hardening.

String ops (str.upper/.lower/.strip, .turbofastapply.str.contains()/ .replace()) are implemented and correct but not faster in practice — benchmarked from 500 to 1,000,000 rows, consistently ~0.6–0.8x plain pandas — so they're excluded from the default engine="auto" and only reachable via explicit engine="native". See "Options" below.

Development

The native extension is built with PyO3 + maturin.

python -m venv .venv
.venv/Scripts/activate        # Windows
pip install maturin pytest pandas numpy
maturin develop                # builds the extension and installs it in the venv
pytest -v

If maturin develop fails to launch on your machine (e.g. maturin.exe can't load api-ms-win-crt-*.dll because only the GNU Rust toolchain is installed and the MSVC Visual C++ Redistributable is missing), use the fallback build script instead. It calls cargo build directly, drops the resulting DLL into the package as a .pyd, and links python/ into the venv via a .pth file (same end result as maturin develop, so import turbofastapply works from anywhere without setting PYTHONPATH):

./build.ps1
.venv/Scripts/python.exe -m pytest -v

CI (GitHub Actions, .github/workflows/ci.yml) runs on Ubuntu with a properly provisioned toolchain, so maturin develop works there regardless.

Usage

The accessor is directly callable — s.turbofastapply(func) is the primary API, not s.turbofastapply.apply(func) (kept only as an alias):

import pandas as pd
import turbofastapply  # registers the .turbofastapply accessor

s = pd.Series([1, 2, 3])
s.turbofastapply(lambda x: x * 2)   # identical result to s.apply(lambda x: x * 2)

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
df.turbofastapply(lambda row: row["a"] + row["b"], axis=1)

# Whitelisted string methods and regex ops (.turbofastapply.str mirrors
# pandas' own .str accessor) are implemented and correct, but not
# faster than plain pandas in practice — see the Status note above.
# They use plain pandas by default; engine="native" forces the compiled
# path if you want it anyway:
names = pd.Series(["  Ada  ", "GRACE", "margaret"] * 50)
names.turbofastapply(str.upper, engine="native")
names.turbofastapply.str.contains(r"^GRACE$")

# .groupby(...).turbofastapply(func) also works — a drop-in for
# .groupby(...).apply(func) with the same engine/verbose/progress_bar
# options as everything else. engine="auto" tries a threaded parallel
# fallback first (measured, not assumed — same approach as the
# GIL-releasing-callable case above, just chunked by group), for
# GroupBy objects with at least 800 groups — a real, ~2x+ speedup for
# GIL-releasing callables, verified never to regress a CPU-bound one
# even at that scale (see claude.md's P9 section for why the threshold
# is that high). Below that it falls back to plain pandas immediately,
# no measurement overhead paid. There's no *native* (Rust) GroupBy path
# though, so engine="native" raises rather than pretending to
# accelerate something that doesn't exist yet.
df.groupby("a").turbofastapply(lambda g: g["b"].sum())

Options

s.turbofastapply(func, engine="auto")     # default: try the native fast path, fall back silently
s.turbofastapply(func, engine="pandas")   # always plain pandas .apply(), skips the eligibility check
s.turbofastapply(func, engine="native")   # require the fast path; raises ValueError with the
                                     # specific reason if func isn't eligible, instead of
                                     # silently falling back

s.turbofastapply(func, verbose=True)      # prints which engine was used and why, to stderr

s.turbofastapply(func, progress_bar=True) # progress bar for the pandas-fallback path only —
                                     # the native path is one vectorized call, nothing to
                                     # report progress on there

Benchmarks

examples/benchmark.py compares plain pandas.apply() against .turbofastapply() on 1,000 rows (median of 50 runs, 5 warmup): the numeric-transform case (x * 2 + 1) lands at ~6.2x and the row-wise case (row['a'] + row['b'], notoriously slow in vanilla pandas since it constructs a Series per row) at ~4.6x. It also runs the string case both ways — default engine="auto" (matches plain pandas exactly, no native path picked) and forced engine="native" — to show directly why "auto" never picks it.

Benchmarks: previous (PyPI 0.1.0) vs now

The 0.1.0 release published to PyPI had a real regression from small data up through a few hundred rows — turbofastapply was slower than plain pandas in that range, not just less of a win. A follow-up optimization pass (not yet released) fixed the redundant work causing it: deduplicated array/Index construction that used to run twice per call, np.isclose()/np.round() scalar calls replaced with plain Python arithmetic in the hot verification path (measured at ~38% of decide()'s total time by itself), and eliminating a redundant DataFrame column-selection + a redundant .isna() column scan in the row-wise path. Median of 100 runs (60 for row-wise), 15 warmup, timed against turbofastapply==0.1.0 installed from PyPI in a clean venv vs. the same numbers from this optimization applied locally:

Numeric path (s.turbofastapply(lambda x: x * 2 + 1))

Row-wise path (df.turbofastapply(lambda row: row['a'] + row['b'], axis=1))

Three passes. Pass 1 (dedup'd Index construction, reused used-column arrays) narrowed the regression but never closed it. Pass 2 raised MIN_ROWS from 50 to 300 so engine="auto" would decline rather than engage a losing native call below that size — parity, not a win. Pass 3 replaced the actual cost driver — a real pandas Series built per probe and verification row — with _RowView, a plain-dict stand-in supporting row['col']/row.col (everything an affine row function needs) without pandas' Series.__init__ overhead; a function needing more of the real Series API (row.sum(), .values, ...) safely declines instead of breaking. That cut the fixed cost 3-4x and let MIN_ROWS drop to 100 — the first size where every trial in an 8-trial sweep cleared 1x:

engine="native" still works below 100 rows for anyone who wants the fast path regardless of profitability (enforce_min_rows=False bypasses the threshold on an explicit request, same as the numeric path). See claude.md for the full investigation.

cargo bench (benches/native_benches.rs) benchmarks the pure Rust compute cores directly (1,000 / 50,000 / 200,000 elements), independent of Python/PyO3 marshaling overhead — useful for catching a regression in the Rust layer itself.

examples/benchmark_vs_competitor.py adds a comparison against another accelerated-.apply() library. It's a separate opt-in script (not a turbofastapply dependency), since that library pulls in dask, tqdm, etc. As of this writing its 1.4.0 release doesn't run cleanly against recent pandas/Python — see that script's error message for the specific compatibility gap if you hit it.