arnio 1.1.0

C++ accelerated CSV preprocessing and data cleaning for pandas

Your CSV hits C++ before Python even wakes up.

Arnio is a compiled C++ data cleaning engine that slots in before pandas.
It parses, infers types, strips whitespace, deduplicates, and normalizes —
all natively, in columnar memory — then hands you a pristine DataFrame.
No .apply(). No lambda chains. No spaghetti.

         

pip install arnio

Quickstart · Why Arnio · Architecture · Benchmarks · Contribute

---

⚡ Quickstart

Three lines. That's the entire workflow.

import arnio as ar

# Load CSV directly through C++ — no Python parsing overhead
frame = ar.read_csv("messy_sales_data.csv")

# Declare what clean data looks like — arnio handles the rest
clean = ar.pipeline(frame, [
    ("strip_whitespace",),
    ("normalize_case", {"case_type": "lower"}),
    ("fill_nulls", {"value": 0.0, "subset": ["revenue"]}),
    ("drop_nulls",),
    ("drop_duplicates",),
])

# Out comes a standard pandas DataFrame — use it like you always have
df = ar.to_pandas(clean)

Every step above executes in C++. Your Python code is a configuration — not the execution engine.

📸 Peek at a 100 GB file without loading it

scan_csv reads only the header + a sample to infer the schema. Zero data loaded.

schema = ar.scan_csv("100GB_file.csv")
# {'id': 'int64', 'name': 'string', 'is_active': 'bool', 'revenue': 'float64'}

Useful for exploring datasets before committing memory.

🧩 Add custom steps without touching C++

Register any Python function as a pipeline step. It receives a DataFrame, returns a DataFrame.

def remove_outliers(df, column="revenue", threshold=100_000):
    return df[df[column] <= threshold]

ar.register_step("remove_outliers", remove_outliers)

# Now use it in any pipeline alongside native C++ steps
clean = ar.pipeline(frame, [
    ("strip_whitespace",),
    ("remove_outliers", {"column": "revenue", "threshold": 50000}),
    ("drop_duplicates",),
])

Custom steps run through a pandas↔ArFrame conversion bridge. Prototype in Python, then optionally migrate hot paths to C++ for full speed.

---

🔍 Why Arnio exists

Every data project starts the same way:

df = pd.read_csv("data.csv")              # 💥 RAM spike — entire file as raw strings
df.columns = df.columns.str.strip()        # Why is this not automatic?
df["name"] = df["name"].str.strip()        # Python loop over every cell
df["name"] = df["name"].str.lower()        # Another Python loop
df = df.dropna()                           # Another pass
df = df.drop_duplicates()                  # Another pass

Six lines. Four full-data passes. All in interpreted Python. This is fine for a Jupyter demo — but it doesn't scale, it doesn't compose, and it definitely doesn't belong in production.

Arnio intercepts this entire pattern. It moves the heavy lifting to C++, replaces imperative chains with a declarative pipeline, and gives you a clean DataFrame in one shot.

Without Arnio df = pd.read_csv(path) df.columns = df.columns.str.strip() for col in str_cols: df[col] = df[col].str.strip() df[col] = df[col].str.lower() df = df.dropna(subset=["revenue"]) df = df.drop_duplicates() # 6+ lines, multiple passes, pure Python

With Arnio frame = ar.read_csv(path) df = ar.to_pandas(ar.pipeline(frame, [ ("strip_whitespace",), ("normalize_case", {"case_type": "lower"}), ("drop_nulls", {"subset": ["revenue"]}), ("drop_duplicates",), ])) # Declarative. Single pipeline. C++ execution.

---

🏗️ Architecture

Arnio is not a pandas wrapper. It's a separate runtime with its own data model.

┌──────────────────────────────────────────────────────────────┐
│  Your Python Code                                            │
│  frame = ar.read_csv("data.csv")                             │
│  clean = ar.pipeline(frame, [...])                           │
│  df = ar.to_pandas(clean)                                    │
└────────────────────────┬─────────────────────────────────────┘
                         │  pybind11 boundary
┌────────────────────────▼─────────────────────────────────────┐
│  C++ Runtime  (_arnio_cpp)                                   │
│                                                              │
│  ┌─────────────┐  ┌─────────────────┐  ┌──────────────────┐ │
│  │  CsvReader   │  │  Frame/Column   │  │  Cleaning Engine │ │
│  │  • RFC 4180  │  │  • Columnar     │  │  • drop_nulls    │ │
│  │  • BOM strip │  │  • std::variant │  │  • fill_nulls    │ │
│  │  • Type      │  │  • Bool null    │  │  • drop_dupes    │ │
│  │    inference │  │    masks        │  │  • strip_ws      │ │
│  │  • Quoted    │  │  • O(1) column  │  │  • normalize     │ │
│  │    fields    │  │    lookup       │  │  • rename/cast   │ │
│  └─────────────┘  └─────────────────┘  └──────────────────┘ │
│                                                              │
│  to_pandas() ──→ zero-copy NumPy buffer (numerics/bools)     │
└──────────────────────────────────────────────────────────────┘

Design decisions that matter

Decision	What it means
Columnar storage	Data lives in typed std::vectors — vector<int64_t>, vector<double>, vector<string> — not rows of variants. Cache-friendly and SIMD-ready.
Boolean null masks	Nulls are tracked in a separate vector<bool>, keeping data vectors dense. No sentinel values, no NaN tricks.
Two-pass CSV read	Pass 1 infers types across all rows. Pass 2 parses values directly into the correct typed column. No string→object→cast overhead.
Zero-copy bridge	to_pandas() exposes C++ memory directly via NumPy's buffer protocol. Numeric and boolean columns cross the boundary without copying.
Step registry	Pipeline steps map to C++ function pointers. Adding a new cleaning primitive is a single function + one registry entry.

Full architecture documentation: ARCHITECTURE.md

---

🏎️ Benchmarks

Setup: Ubuntu, Python 3.12, 1M rows × 12 columns, synthetic messy CSV.
Reproduce: make benchmark — generates data and runs both engines.

                     pandas         arnio
────────────────────────────────────────────
Exec Time (avg)       4.73s         5.75s
Peak RAM               211MB         212MB
API Clarity         Imperative    Declarative

Arnio is near memory parity in the reference benchmark while replacing ad-hoc Python string loops with a compiled, declarative pipeline. Validate memory and speed on your own workload. The execution time gap is a known, active optimization target — the current drop_duplicates and strip_whitespace implementations use unoptimized row-key serialization.

✅ What's already won	🎯 What's being optimized
Native C++ parsing eliminates Python memory spikes Columnar storage matches pandas' internal efficiency Declarative API eliminates .apply() spaghetti Zero-copy bridge for numeric conversions	drop_duplicates — replace string serialization with hash-based comparisons strip_whitespace — in-place mutation instead of copy-on-write Parallel column processing via std::thread Help close the gap →

---

🧰 Cleaning primitives

Every operation below runs natively in C++. No Python loops.

Primitive	What it does	Example
drop_nulls	Remove rows with null/empty values	ar.drop_nulls(frame, subset=["age"])
fill_nulls	Replace nulls with a scalar	ar.fill_nulls(frame, 0, subset=["revenue"])
drop_duplicates	Deduplicate rows (first/last/none)	ar.drop_duplicates(frame, keep="first")
strip_whitespace	Trim leading/trailing spaces from strings	ar.strip_whitespace(frame)
normalize_case	Force lower/upper/title case	ar.normalize_case(frame, case_type="title")
rename_columns	Rename columns via mapping	ar.rename_columns(frame, {"old": "new"})
cast_types	Cast column types	ar.cast_types(frame, {"age": "int64"})
clean	Convenience shorthand	ar.clean(frame, drop_nulls=True)

Or compose them all into a pipeline:

clean = ar.pipeline(frame, [
    ("strip_whitespace",),
    ("normalize_case", {"case_type": "lower"}),
    ("fill_nulls", {"value": "unknown", "subset": ["city"]}),
    ("drop_duplicates", {"keep": "first"}),
])

---

🧠 Data quality engine

Arnio now includes built-in dataset understanding before you analyze in pandas.

report = ar.profile(frame)
print(report.summary())

suggestions = ar.suggest_cleaning(frame)
clean = ar.pipeline(frame, suggestions)

For production data contracts:

schema = ar.Schema({
    "id": ar.Int64(nullable=False, unique=True),
    "email": ar.Email(nullable=False),
    "revenue": ar.Float64(nullable=True, min=0),
})

result = ar.validate(frame, schema)
if not result.passed:
    print(result.to_pandas())

For low-risk automatic cleanup:

clean, report = ar.auto_clean(frame, mode="strict", return_report=True)

This is the layer pandas does not try to own: profiling, data contracts, row-level validation issues, and safe cleaning suggestions for messy incoming datasets.

---

🗺️ Roadmap

Version	Focus	Status
v1.0	Stable release · cross-platform wheels · CI/CD · PyPI publishing · Google Colab support	✅ Shipped
v0.2	C++ pipeline optimization · speed parity with pandas · hash-based deduplication	🔨 Active
v0.3	Chunked / streaming processing · Parquet & JSON readers	📋 Planned
v0.4	Parallel column processing · SIMD string operations	💭 Exploring

---

🤝 Contribute

Arnio is a GSSoC 2026 project with 55+ open issues across all skill levels.

You don't need C++ to contribute

Most new features are pure Python pipeline steps:

# 1. Write a function that takes a DataFrame and returns a DataFrame
def remove_special_chars(df, columns=None):
    cols = columns or df.select_dtypes("object").columns
    for col in cols:
        df[col] = df[col].str.replace(r"[^a-zA-Z0-9\s]", "", regex=True)
    return df

# 2. Register it
ar.register_step("remove_special_chars", remove_special_chars)

# 3. Write tests, open a PR. That's it.

If you do know C++

The biggest performance wins are in:

• drop_duplicates — replacing std::ostringstream row serialization with proper hash-based comparisons
• strip_whitespace — converting from copy-on-write to in-place mutation
• Parallel column processing — std::thread across independent columns

Getting started

# macOS / Linux
git clone https://github.com/im-anishraj/arnio.git && cd arnio
make install   # pip install -e ".[dev]" + pre-commit
make test      # pytest with coverage
make lint      # ruff + black

# Windows
pip install -e ".[dev]"
pre-commit install
pytest tests/ -v

PR titles must follow Conventional Commits — feat:, fix:, docs:, chore:. Our release pipeline auto-generates changelogs from these.

📖 Full Contributing Guide ·  🐛 Open Issues ·  💬 Discussions

---

📐 Project structure

arnio/
├── cpp/
│   ├── include/arnio/      # C++ headers — types, column, frame, csv_reader, cleaning
│   └── src/                 # C++ implementations (~30 KB of compiled logic)
├── bindings/
│   └── bind_arnio.cpp       # pybind11 module — the Python↔C++ bridge
├── arnio/
│   ├── __init__.py          # Public API surface
│   ├── io.py                # read_csv, scan_csv
│   ├── cleaning.py          # Python wrappers for C++ cleaning functions
│   ├── pipeline.py          # Step registry + pipeline executor
│   ├── convert.py           # to_pandas (zero-copy), from_pandas
│   ├── frame.py             # ArFrame — lightweight C++ Frame wrapper
│   └── exceptions.py        # ArnioError, UnknownStepError, CsvReadError, TypeCastError
├── tests/                   # pytest suite — CSV, cleaning, pipeline, conversions
├── benchmarks/              # Reproducible arnio vs pandas benchmark
├── examples/                # basic_usage.py, custom_step.py
└── website/                 # Project website — arnio.vercel.app

---

Stop writing cleaning scripts. Declare clean data.

     

_{Built with C++ and pybind11 · Licensed under MIT · Maintained by @im-anishraj}