xsxl 0.1.0

High-performance Python library for streaming Excel (XLSX) data to Apache Arrow format

xsxl - High-Performance Excel to Arrow Parser

A blazingly fast Python library for streaming Excel (XLSX) data directly to Apache Arrow format, built in Rust with PyO3.

Features

• Streaming Architecture: Process files larger than memory with configurable batch sizes
• Lazy Loading: Only loads metadata on open, data parsed on-demand
• Type Inference: Automatically infers Arrow types from Excel number formats
• High Performance: ~3M cells/sec with full type inference
• Zero-Copy: Uses Arrow C Data Interface for efficient data transfer
• Thread-Safe: GIL released during parsing for true parallelism

Installation

pip install xsxl

Quick Start

import xsxl

# Open workbook (metadata only, no data loaded)
wb = xsxl.open("data.xlsx")

# Iterate over sheets
for sheet_name in wb:
    print(f"Found sheet: {sheet_name}")

# Get a specific sheet (still no data loaded)
sheet = wb["Sales"]

# Reference a range (still lazy)
range = sheet.get_range("A1:Z10000")

# Now data is parsed and streamed
for batch in range.iter_batches(batch_size=5000):
    print(f"Got {batch.num_rows} rows")

# Or convert to pandas/polars
df = range.to_pandas()
pl_df = range.to_polars()

Usage Examples

Stream Large Files

wb = xsxl.open("huge_file.xlsx")
range = wb["Sheet1"].get_range("A1:ZZ1000000")

# Process in batches to keep memory bounded
for batch in range.iter_batches(batch_size=10000):
    # Process each batch
    process_batch(batch.to_pandas())

Load Multiple Sheets Selectively

wb = xsxl.open("multi_sheet.xlsx")

# Load only sheets matching pattern
data = {}
for name in wb:
    if name.startswith("Sales_"):
        data[name] = wb[name].get_range("A1:Z1000").to_pandas()

Transpose Mode

# For data laid out horizontally
range = sheet.get_range("A1:J100", transpose=True)
df = range.to_pandas()  # Columns become rows

Performance

Benchmarked on 2024 MacBook Pro M4:

• Parsing: 2,989,014 cells/sec with full type inference
• Memory: <100MB for metadata, configurable batch sizes for data
• GIL: Released during parsing for true parallelism

License

MIT OR Apache-2.0