vroom-csv 0.1.0

High-performance CSV parser with SIMD acceleration

vroom-csv

High-performance CSV parser with SIMD acceleration for Python.

Features

• SIMD-accelerated parsing using Google Highway for portable vectorization
• Multi-threaded parsing for large files
• Automatic dialect detection (delimiter, quoting, line endings)
• Automatic type inference detects integers, floats, booleans, and strings
• Arrow PyCapsule interface for zero-copy interoperability with PyArrow, Polars, DuckDB
• Full type annotations with IDE autocomplete support

Installation

pip install vroom-csv

For Arrow interoperability:

pip install vroom-csv[arrow]   # PyArrow support
pip install vroom-csv[polars]  # Polars support

For development:

pip install vroom-csv[dev]

Quick Start

import vroom_csv

# Read a CSV file
table = vroom_csv.read_csv("data.csv")

print(f"Loaded {table.num_rows} rows, {table.num_columns} columns")
print(f"Columns: {table.column_names}")

# Access data
names = table.column("name")
first_row = table.row(0)

Arrow Interoperability

vroom-csv implements the Arrow PyCapsule interface for zero-copy data exchange:

PyArrow

import pyarrow as pa
import vroom_csv

table = vroom_csv.read_csv("data.csv")
arrow_table = pa.table(table)  # Zero-copy conversion

# Now use PyArrow's features
arrow_table.to_pandas()

Polars

import polars as pl
import vroom_csv

table = vroom_csv.read_csv("data.csv")
df = pl.from_arrow(table)  # Zero-copy conversion

# Now use Polars' features
df.filter(pl.col("age") > 30)

DuckDB

import duckdb
import vroom_csv

table = vroom_csv.read_csv("data.csv")
result = duckdb.query("SELECT * FROM table WHERE age > 30")

API Reference

read_csv(path, ...)

Read a CSV file and return a Table object.

read_csv(
    path: str,
    delimiter: str | None = None,
    quote_char: str = '"',
    has_header: bool = True,
    skip_rows: int = 0,
    n_rows: int | None = None,
    usecols: list[str | int] | None = None,
    dtype: dict[str, str] | None = None,
    null_values: list[str] | None = None,
    empty_is_null: bool = True,
    encoding: str = "utf-8",
    num_threads: int = 1,
) -> Table

Parameters:

• path: Path to the CSV file
• delimiter: Field delimiter (auto-detected if None)
• quote_char: Quote character for quoted fields
• has_header: Whether the first row contains column headers
• skip_rows: Number of data rows to skip after the header
• n_rows: Maximum number of rows to read (None = all)
• usecols: List of column names or indices to read
• dtype: Dict mapping column names to types ("string", "int", "float", "bool")
• null_values: List of strings to treat as null values
• empty_is_null: Whether empty fields are treated as null
• encoding: File encoding (currently UTF-8 only)
• num_threads: Number of threads for parsing

detect_dialect(path)

Detect CSV dialect (delimiter, quoting, line endings).

dialect = vroom_csv.detect_dialect("data.csv")
print(dialect.delimiter)   # ','
print(dialect.quote_char)  # '"'
print(dialect.has_header)  # True
print(dialect.confidence)  # 0.95

Table

The Table class represents parsed CSV data.

Properties:

• num_rows: Number of data rows (excluding header)
• num_columns: Number of columns
• column_names: List of column names

Methods:

• column(index_or_name): Get column data as list
• row(index): Get row data as list
• has_errors(): Check if any parse errors occurred
• error_summary(): Get a summary of parse errors
• errors(): Get list of parse error messages

Arrow PyCapsule Protocol:

• __arrow_c_schema__(): Export schema via Arrow C Data Interface
• __arrow_c_stream__(): Export data via Arrow C Stream Interface

Examples

See the examples directory for Jupyter notebooks demonstrating:

• Getting Started - Basic usage
• Arrow Interoperability - PyArrow, Polars, DuckDB

Benchmarks

Run performance benchmarks against pandas, Polars, PyArrow, and DuckDB:

cd benchmarks
pip install pandas polars pyarrow duckdb
python benchmark_csv.py --sizes 1,10,100

See benchmarks/README.md for details.

License

MIT License - see LICENSE file in the repository root.