🚀 MXFrame

GPU-accelerated DataFrames — Python ergonomics, Mojo AOT kernels.

MXFrame is a DataFrame query engine that pairs a Polars-style Python API with pre-compiled Mojo AOT kernels. The current GPU build is validated on NVIDIA; AMD and Apple Silicon validation remain on the roadmap.

✨ Why MXFrame?

MXFrame uses a portable Mojo kernel architecture rather than CUDA-specific source. Kernels compile once to shared libraries and dispatch directly at runtime.

What changed in v0.4 and v0.5

• v0.4.0 — correct string/composite joins: shared dictionaries assign identical dense IDs across both join sides; mixed composite keys and SQL null semantics are preserved.
• v0.5.0 — native GPU UTF-8 predicates: startswith, literal contains, string equality/inequality, and packed non-null literal isin operate directly on Arrow offsets and byte buffers.
• v0.5.0 — weak-query fusion: dedicated compact GPU paths remove expanded intermediates from Q4, Q6, Q13, and Q21.
• v0.5.0 — bounded benchmarking and visualization: every query/engine runs in an isolated process; Q4, Q6, Q12, Q13, Q14, and Q21 have step-by-step kernel visualizations.

See v0.5.0 release notes for measured gains, limitations, and the parity roadmap.

⚡ Quick Start

# 1. Install pixi (Modular's package manager)
curl -fsSL https://pixi.sh/install.sh | bash

# 2. Clone and set up
git clone https://github.com/abhisheksreesaila/mxframe
cd mxframe
pixi install

# 3. Verify GPU is working
pixi run python3 scripts/_check_gpu.py

import pyarrow as pa
from mxframe import LazyFrame, Scan, col, lit

data = pa.table({
    "dept":   pa.array(["eng", "eng", "mkt", "mkt", "eng"]),
    "salary": pa.array([120.0, 95.0, 80.0, 110.0, 130.0], pa.float32()),
    "age":    pa.array([32, 28, 35, 29, 40], pa.int32()),
})

result = (
    LazyFrame(Scan(data))
    .filter(col("age") > lit(28))
    .groupby("dept")
    .agg(
        col("salary").sum().alias("total_salary"),
        col("age").count().alias("headcount"),
    )
    .sort(col("total_salary"), descending=True)
    .compute(device="gpu")   # or "cpu"
)
print(result.to_pandas())

  dept  total_salary  headcount
0  eng         345.0          3
1  mkt         110.0          1

📊 Performance — bounded v0.5.0 benchmark

RTX 3090, warm median of three runs. Every query and engine ran in its own process so Arrow, Pandas, and CUDA allocations were released between measurements.

• 1M: MX CPU beat Polars on 22/22; MX GPU beat Polars on 15/17 comparable paths.
• 10M: MX CPU beat Polars on 18/22; MX GPU beat Polars on 14/17 comparable paths.
• Q8 GPU remains N/A:JIT; Q15, Q17, Q18, and Q20 GPU workers exceeded the bounded timeout.
• cuDF was unavailable on this machine, so RAPIDS remains N/A and no RAPIDS parity claim is made.

1M representative queries

10M representative queries

Full matrices: 1M CSV · 10M CSV · methodology and kernel catalogue

What remains for parity

Performance parity: replace Q8's JIT path; fix Q15/Q17/Q18/Q20 GPU timeouts; improve Q1 and Q14 GPU execution; run synchronized 1M/10M measurements on a compatible RAPIDS/cuDF environment.

Feature parity: native GPU UTF-8 dictionary construction and string/date gather; GPU windows; semi/anti/as-of/range joins; broader string/regex, datetime, nested-type, reshape, SQL, streaming, and I/O pushdown coverage.

📚 Docs

📦 Dependencies

📄 License

Apache 2.0 — see LICENSE.