fixedwidthparser

Read and write fixed-width files from Python through FixedWidthParser's NativeAOT library: no .NET runtime, parallel reads, results as NumPy views, Arrow, pandas or polars.

pip install "fixedwidthparser[numpy]"    # fw.read: NumPy views
pip install "fixedwidthparser[pandas]"    # or [pyarrow], [polars]

Wheels ship for Windows x64, macOS (arm64, x64) and Linux (glibc and musl, x64 and arm64). The package installs as fixedwidthparser and imports as fixedwidth. pyarrow has no musllinux wheels, so on Alpine and other musl systems Arrow, pandas, polars and writing need a pyarrow built for them.

import fixedwidth as fw

# B3's COTAHIST quote history: 245-byte lines, a header (00) and a trailer (99) around the quotes (01).
layout = fw.Layout([
    fw.Column("tipreg", 0, 2, fw.String),
    fw.Column("data_pregao", 2, 8, fw.Date, format="yyyyMMdd"),
    fw.Column("codneg", 12, 12, fw.String),
    fw.Column("preabe", 56, 13, fw.Float64, scale=2),   # N(11)V99: two implied decimals
    fw.Column("quatot", 152, 18, fw.Int64),
])
options = fw.ReadOptions(header_lines=1, skip_invalid=True)   # the trailer fails to parse and is skipped

df = fw.read_pandas("COTAHIST_A2023.TXT", layout, options)

Reading

source is a path, bytes/bytearray/memoryview/io.BytesIO to read from memory (a BytesIO in place, from its position), or any stream with readinto() or read(): an open file, a pipe, gzip.open(...), zipfile.ZipFile(...).open(...). read_batches and read_arrow_stream pull from a stream as the batches are taken, in constant memory; the whole-table reads read it into memory first, since they split it into byte ranges.

import zipfile

with zipfile.ZipFile("COTAHIST_A2023.ZIP") as archive, archive.open("COTAHIST_A2023.TXT") as stream:
    for batch in fw.read_batches(stream, layout, fw.ReaderOptions(read=options, batch_rows=100_000)):
        ...

Every optional setting lives in fw.ReadOptions: header_lines, skip_invalid, degree_of_parallelism (0 = one thread per processor, 1 = sequential), encoding (a code page or "latin-1", "cp1252", ...), max_line_length and skip_byte_order_mark. A parallel read gives the same rows, in the same order, as a sequential one.

A line that fails to parse raises fw.ParseError with line_number and column, unless skip_invalid is set.

fw.Table maps a column name to a NumPy array (a list for strings), and has num_rows, column_names, lines_read, lines_skipped, validity(name) (a bool array, True where the row holds a value, or None when the column has no nulls) and to_arrow(), which shares the native buffers. Without pyarrow, read_polars reads sequentially through the Arrow stream.

Threads and asyncio

Every function releases the GIL for its native work, so reads and writes on several threads, or through asyncio.to_thread, run in parallel:

frames = await asyncio.gather(*(asyncio.to_thread(fw.read_polars, path, layout, options) for path in paths))

• A fw.Layout is immutable and can be shared by every thread at once.
• A failure is raised on the thread whose call failed, with that call's message and line number.
• A read_batches iterator, and the stream it reads, belong to one thread at a time.
• A stream's read() is called on the thread doing the read and takes the GIL while it runs; Python's files, sockets and zlib release it again while they wait or decompress, so other threads keep going.
• degree_of_parallelism=0 gives each call one thread per processor. When many calls already run at once, pass 1: the calls are the parallelism, and splitting each one too only makes them queue for the same cores.

Columns

fw.Column(name, start, length, type, *, nullable, scale, format, trim_char, trim_mode, alignment, padding, overflow, dictionary). Positions are bytes. Types: fw.String, fw.Int64, fw.Float64, fw.Decimal (int64 unscaled, Arrow decimal128(18, scale)), fw.Date, fw.Time, fw.Timestamp. scale is the number of implied decimals: 0000000123456 with scale=2 reads as 1234.56. format uses .NET date syntax (yyyyMMdd, HHmmss). trim_mode, alignment and overflow take fw.TrimMode, fw.Alignment and fw.Overflow.

dictionary=True on a fw.String column hands it to Arrow dictionary-encoded, so read_pandas gives a category column and read_polars a Categorical one: for a column of few distinct values (a market code, a currency, a record type) far less memory, and faster group-bys. The encoding costs a pass over the column, about 3 ms per million rows per column, run in parallel across columns. fw.read's Table keeps plain strings, and its to_arrow() encodes the same columns. Categorical columns write like strings.

fw.Layout(columns, fw.LayoutOptions(line_length=..., filler=...)) sets how written lines are laid out: line_length past the last column pads each line, and filler (default a space) fills the gaps between columns.

Writing

fw.write(df, "out.txt", layout, fw.WriteOptions(degree_of_parallelism=0))
data = fw.write_bytes(df, layout)

with gzip.open("out.txt.gz", "wb") as stream:   # or an open file, a socket's makefile("wb"), ...
    fw.write(df, stream, layout)

fw.write takes a path or a binary stream with write(). A stream gets the lines in order, a batch per call on the calling thread, while the next batches format in parallel, so memory stays at a batch per thread; writing to an open file is as fast as writing to its path. For the bytes themselves, write_bytes beats an io.BytesIO, which copies itself each time it grows.

Takes pandas, polars, pyarrow or anything with __arrow_c_stream__ (needs pyarrow), matching columns by name. padding="0" writes zero-padded numbers with the sign leftmost; a null writes as its column filled with trim_char. Parallel writes are byte-identical to serial ones. fw.WriteOptions also takes encoding, newline (default "\n") and batch_rows.

A fw.Decimal column stores its value times 10**scale as an int64. Decimals and integers are scaled exactly, and a decimal with more digits than scale raises. Floats round half to even to scale. A value that is NaN, infinite or outside int64 once scaled raises before anything is written.

Building from source

python python/scripts/build_native.py      # requires the .NET 10 SDK; publishes the library into src/fixedwidth/_lib
pip install -e "python[dev,pandas,polars]"
pytest python/tests

To point at a library built elsewhere, set FIXEDWIDTH_NATIVE_LIB to its full path.