rdsframe 0.4.0b2

Fast, memory-conscious reader for RDS data.frames

rdsframe

rdsframe reads common R data.frame and data.table objects from binary RDS files directly into pandas. Numeric vectors are filled in their final NumPy allocation, which avoids the large intermediate Python object tree created by general-purpose R serialization readers.

Status: 0.4.0b2 pre-release (beta). Validate results against R for critical pipelines. Unsupported R structures fail explicitly; they are never silently coerced.

Features

• R serialization versions 2 and 3, XDR and native binary formats.
• Uncompressed, gzip, bzip2, xz, and zstd containers (zstd is what R >= 4.5 writes for compress = "zstd"; reading it needs Python >= 3.14 or the rdsframe[zstd] extra).
• Integer, double, logical, character, factor (ordered and unordered), Date, POSIXct, POSIXlt, difftime, raw, and the ALTREP representations R actually serializes (compact integer/real sequences, deferred as.character strings, sort()/attribute wrappers). Complex columns become complex pandas series or STRUCT(real, imag) in Parquet.
• Batched string parsing: STRSXP columns are decoded (or structurally skipped) from large chunks instead of three stream reads per element, which is what makes text-heavy catalogs practical (see BENCHMARKS.md).
• Optional compiled scanner: Cython accelerates only structural CHARSXP skipping when the extension is available; materializing strings remains on the tested Python path, as does all parsing without the module.
• Explicit policies for time zones, invalid timestamps, and heterogeneous list-columns; lossy conversion is never activated implicitly.
• A single data.frame or a named list of data.frame objects -- and, via read_r_object() or rdsframe dump, any other supported R value: plain lists, nested structures, matrices, S4 objects (as a dict of slots), and environments (as a dict of their contents).
• Reads from a path, raw bytes, or any seekable binary stream.
• Character row names become the pandas index; R's compact default row numbering stays a RangeIndex.
• Nullable pandas integer/boolean dtypes and categorical factors.
• Public Arrow-table reads plus Parquet export through PyArrow or the optional memory-bounded DuckDB engine, and a command-line interface.
• Deferred open_rds() datasets with structural schemas and column projection, exact/metadata-only inspection modes, and Arrow-backed Polars/DuckDB adapters.
• Configurable defensive limits for untrusted or unexpectedly large inputs.
• Low-allocation table catalogs and selective extraction by index, name, or (for read_rds()) column, plus materialize_uncompressed() for repeated seek-based access to large compressed files.
• Validated against a golden-file corpus written by R 4.5.0 itself (tests/data/r450, generated by tests/data/gen_fixtures.R).

RData workspaces, ASCII serialization, closures/language objects, and custom third-party ALTREP classes are intentionally unsupported -- and fail with an error that names the R type, so applications can tell users exactly why a slower general-purpose fallback is being used.

How it compares

One fresh process per read on the same machine (Windows 11, Python 3.12, 2026-07-16), wall time / peak RSS, against pyreadr 0.5.6 (librdata, C) and rdata 1.1.0. Synthetic inputs are uncompressed; details, versions, and caveats in BENCHMARKS.md -- treat these as one data point, not a universal claim.

Input	rdsframe	pyreadr	rdata
2M rows x 8 numeric columns (88 MiB)	0.7 s / 199 MB	1.8 s / 469 MB	1.3 s / 490 MB
1M rows x 5 text-heavy columns	2.6 s / 228 MB	16.9 s / 418 MB	39.3 s / 1.7 GB
nflverse play-by-play 2023 (49,665 x 372, gzip)	4.3 s / 322 MB	4.7 s / 766 MB	94 s / 3.7 GB
123 MiB gzip, root = named list of 6 data.frames	32 s (all tables)	returns {} silently	impractical
Catalog only (list_rds_tables) of that same file	3.9 s	n/a	n/a

Two structural differences matter more than the timings. pyreadr (librdata) cannot read an RDS whose root is a list of data.frames -- it returns an empty dict without raising -- while that layout is exactly what rdsframe targets, including selective extraction. And where all three readers can read the same file, rdsframe's output was verified value-for-value against pyreadr (372/372 columns identical on the nflverse file) and against R itself via checksums.

Install

pip install rdsframe

For Parquet export:

pip install "rdsframe[parquet]"

This installs PyArrow and does not require DuckDB. Install the streaming, column-staged engine when conversion must keep peak memory tied to a column batch rather than a complete table:

pip install "rdsframe[duckdb]"

For Polars:

pip install "rdsframe[polars]"

For zstd-compressed RDS on Python < 3.14:

pip install "rdsframe[zstd]"

Source builds try to compile the small Cython-generated scanner from portable C, but compilation failure is optional and never prevents installation. Check the active installation with rdsframe.compiled_backend_available().

Python API

from rdsframe import read_rds

data = read_rds("measurements.rds")
print(data.head())

A named R list returns dict[str, pandas.DataFrame]. To require the legacy first-table behavior, use read_rds_dataframe().

For text-heavy data that still needs to become a pandas DataFrame, keep strings in Arrow-backed pandas arrays:

data = read_rds("measurements.rds", strings="pyarrow")

To bypass pandas entirely, request public Arrow tables:

from rdsframe import read_rds_arrow

table = read_rds_arrow("measurements.rds")

A named list returns dict[str, pyarrow.Table].

Deferred datasets and inspection

open_rds() creates a lightweight handle. Accessing schema performs or reuses the structural catalog scan; payloads are not converted to NumPy, pandas, Arrow or Polars until a terminal operation is requested:

from rdsframe import open_rds

dataset = open_rds("measurements.rds")

print(dataset.schema)
print(dataset.columns)
print(dataset.shape)

preview = dataset[["peso", "edad"]].head(10)
peso = dataset["peso"].collect()
arrow = dataset.select(["peso", "edad"]).to_arrow()

For a named list of data.frames, choose one explicitly:

stations = open_rds("workspace.rds").table("stations")

Projection is lazy and a single-root data.frame skips unselected columns. head() currently limits the returned pandas result, not the underlying RDS vector read: R serializes complete columns, so the selected columns are still consumed. Compressed input must also be decompressed sequentially to reach a later column.

Metadata-only inspection never materializes column payloads:

from rdsframe import inspect_rds

info = inspect_rds("measurements.rds")
print(info.rows, info.columns, info.compression)
for column in info.tables[0].schema:
    print(column.name, column.logical_type, column.factor, column.estimated_bytes)

Fixed-width columns have a structural memory estimate. Text/list memory and missing counts cannot be known without reading their elements; request the explicit scan when exact Arrow buffer sizes and null counts are needed:

info = inspect_rds("measurements.rds", mode="scan")
print(info.statistics_complete)
print(info.tables[0].schema[0].missing_count)

Polars and DuckDB

Both adapters reuse the Arrow conversion and avoid pandas:

from rdsframe import open_rds, read_rds_polars

frame = read_rds_polars("measurements.rds")
frame = open_rds("measurements.rds").select(["edad", "peso"]).to_polars()

DuckDB can consume the projected table as a relation or registered SQL view:

dataset = open_rds("measurements.rds").select(["sexo", "edad", "peso"])

relation = dataset.to_duckdb()
result = relation.filter("edad >= 18").aggregate("sexo, avg(peso)")

connection = dataset.register_duckdb("measurements")
result = connection.sql("""
    SELECT sexo, avg(peso)
    FROM measurements
    WHERE edad >= 18
    GROUP BY sexo
""")

This is a Python/Arrow registration, not yet a native DuckDB read_rds('path') table function; query projection should therefore be applied on the RDSDataset before registration.

To read only a few fields from a wide single data.frame, select columns by zero-based index or exact name. Unselected columns are structurally skipped: they are never allocated as NumPy arrays or pandas objects.

data = read_rds("measurements.rds", columns=["station", "value", "date"])

columns requires the RDS root to be a single data.frame. For a multi-table RDS, select the table first with extract_rds_tables() or to_parquet(). Selecting by name performs one extra structural pass to resolve names to positions (bounded memory, no payload allocation); integer indices skip that pass entirely.

The saving depends on the type of the skipped columns, not just their count. RDS stores columns sequentially with variable-length encoding: skipping a numeric, logical, or raw column in an uncompressed file is a real seek() (near-zero cost); skipping a character column still visits every row to find where it ends, the same traversal a full read does, so it mainly saves the decode/interning/pandas-object cost rather than the row scan itself. columns is most valuable for RAM (skipped columns are never allocated) and for wide tables with many unwanted numeric fields.

Reading non-tabular RDS files

Not every RDS file is a data.frame. read_rds() raises UnsupportedRDS for those on purpose, since silently coercing a plain list into a table would be worse than failing loudly. Use read_r_object() instead:

from rdsframe import read_r_object

data = read_r_object("workspace.rds")

A named R list becomes a dict; an unnamed list becomes a list; a nested data.frame anywhere inside either still becomes a pandas DataFrame. Atomic vectors get the same type rules as a data.frame column (factor, Date, POSIXct, difftime, matrices via a dim attribute) but come back as a plain Python scalar or list rather than a pandas Series. A named atomic vector (c(a = 1, b = 2)) becomes a dict like a named list does, and a class-less matrix keeps its native NumPy dtype (dimnames are not represented). This is a general, exploratory reader, not the fast path: it materializes the whole structure in memory rather than streaming it, so prefer read_rds() / to_parquet() whenever the file actually is tabular.

from rdsframe import ReaderLimits, read_rds, to_parquet

limits = ReaderLimits(max_vector_length=100_000_000)
data = read_rds("input.rds", limits=limits, strings="string")

tables = to_parquet("input.rds", "output", compression="zstd")

engine="auto" selects DuckDB when installed and otherwise writes with PyArrow. Use engine="pyarrow" to require the dependency-light route or engine="duckdb" to require column-staged, spill-to-disk conversion.

Safety and fidelity policies are explicit:

tables = to_parquet(
    "input.rds",
    "output",
    max_tables=100,            # fail before writing partial results
    max_root_items=10_000,
    posixct_mode="preserve",   # or "utc_naive" for constrained executables
    invalid_timestamp="error", # or "null" to opt into coercion
    list_column_mode="infer",  # or explicit "json" / "string"
)

max_tables never truncates a file. Exceeding a configured limit raises RDSLimitError, cleans temporary data, and leaves no apparently successful partial output.

The staging policy is adaptive. By default, a temporary Parquet contains at most 16 columns or approximately 128 MiB of Arrow buffers, whichever is reached first. This reduces temporary-file count on wide tables while bounding memory:

tables = to_parquet(
    "input.rds",
    "output",
    stage_max_columns=16,
    stage_max_bytes=128 * 1024 * 1024,
    gc_collect_every=16,  # use 0 to disable explicit cyclic GC
)

CLI

rdsframe inspect input.rds
rdsframe list input.rds --cache
rdsframe list input.rds --catalog input.rdsframe.json
rdsframe convert input.rds output/ \
  --engine pyarrow \
  --table-name measurements \
  --catalog input.rdsframe.json \
  --max-tables 100 \
  --list-column-mode json \
  --invalid-timestamp null
rdsframe dump input.rds

Run rdsframe convert --help for staging, memory, time-zone and limit controls.

rdsframe dump prints any supported R object -- not only data.frames -- as an indented tree, so an unknown RDS file can be explored before deciding how to convert it. Deeply nested structures are summarized rather than exploded: --max-items bounds children per node (default 10), --max-depth bounds nesting (default 8), and --json emits the complete object as JSON instead (data.frames appear as {"$r_type": "data.frame", ...} objects).

$ rdsframe dump results.rds --max-items 2 --max-depth 3
<list, 10 items>
  [0] <list, 10 items>
    [0] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    [1] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    ... (+8 more items)
  [1] <list, 10 items>
    ...

List and selectively extract tables

Discover tables without building their pandas, NumPy, Arrow, or Parquet payloads:

from rdsframe import list_rds_tables

catalog = list_rds_tables("workspace.rds", cache=True)
for table in catalog.tables:
    print(table.index, table.name, table.rows, table.columns)

With cache=True, the first scan writes workspace.rdsframe.json; later calls return it immediately while source path, size, and modification time match. Pass an explicit path as cache="catalogs/workspace.json" when desired.

Extract by zero-based index in one pass:

from rdsframe import extract_rds_tables

extract_rds_tables("workspace.rds", "output", [0, 2, 7])

Or select by name while reusing the validated catalog:

from rdsframe import RDSCatalog, extract_rds_tables

catalog = RDSCatalog.load("workspace.rdsframe.json")
extract_rds_tables(
    "workspace.rds",
    "output",
    ["measurements", "stations"],
    catalog=catalog,
)

The catalog stores the absolute path, file size and nanosecond modification time. An explicitly loaded stale or foreign catalog raises RDSCatalogError before conversion; the opt-in automatic cache rebuilds stale sidecars. Name-based to_parquet() / extract_rds_tables() selection automatically creates and reuses this sidecar when no explicit catalog is supplied.

RDS is sequential and normally stores list names after all elements. Therefore:

• listing avoids column allocations and temporary outputs, but compressed input must still be traversed and decompressed;
• uncompressed binary RDS payloads are skipped with direct seeks; compressed payloads are discarded through a reusable bounded buffer;
• integer selection without a catalog converts only chosen tables but may scan the remaining stream to recover original names;
• a validated catalog enables early stop after the last selected table and is the fastest repeated workflow;
• the first name selection without an explicit catalog performs a structural listing pass and saves it; subsequent selections reuse the validated cache.

The source distribution includes BENCHMARKS.md with preliminary reproducible results and their limitations.

Memory model

read_rds() returns pandas objects and therefore requires the resulting table to fit in memory. Numeric columns avoid a full-size temporary bytes buffer.

to_parquet(engine="duckdb") is the lowest-memory path: it constructs Arrow-native columns and stages bounded groups before DuckDB combines them with a positional join. Character vectors are built from offsets, UTF-8 data and validity buffers without retaining a Python str object for every row. Peak parser memory is tied to the current column plus the configured staging batch, not to the entire data frame. DuckDB can spill merge work to disk, controlled by memory_limit and temp_directory. The PyArrow engine and read_rds_arrow() materialize a complete table, trading higher peak memory for fewer dependencies.

For the lowest possible peak, set stage_max_columns=1. For faster conversion of wide, narrow-column tables, increase the batch limits after benchmarking.

The result is directly queryable without loading it in memory:

SELECT category, avg(value)
FROM read_parquet('output/measurements.parquet')
WHERE date >= DATE '2025-01-01'
GROUP BY category;

Development

python -m pip install -e ".[dev,duckdb,polars,zstd]"
pytest
ruff check .
mypy src/rdsframe
pytest --cov=rdsframe --cov-fail-under=80
python -m build
twine check dist/*

Before a public release, run the test matrix and publish to TestPyPI first. The project lives at https://github.com/mmanaylopez/rdsframe.

Releasing

1. Bump the version in pyproject.toml and src/rdsframe/__init__.py, and date the CHANGELOG.md section.
2. Commit, push, and wait for the full CI matrix to pass (it validates the Cython build on the three platforms, the pure-Python fallback, and the minimum supported NumPy/pandas versions -- environments a single dev machine cannot cover).
3. Tag vX.Y.Z and push the tag: release.yml builds the sdist plus binary wheels via cibuildwheel and publishes through PyPI Trusted Publishing (one-time setup: PyPI project settings -> Publishing -> GitHub publisher for this repo, workflow release.yml, environment pypi).
4. Manual fallback: upload only the sdist (python -m build --sdist, then twine upload dist/*.tar.gz). Never upload a locally built wheel: without a working compiler, setuptools still tags it platform-specific even though the accelerator is missing, and that wheel would shadow the sdist for every user of that platform.

RData and fallback integration

rdsframe deliberately rejects .RData/.rda workspaces. Applications that need both formats should inspect the container and use rdsframe for supported binary RDS files, retaining rdata or another general reader as fallback for RData, unsupported ALTREP classes, and richer R objects. An UnsupportedRDS exception is the explicit signal to activate that fallback.

Conversion policies

• POSIXct preserves its R time-zone attribute by default. Applications that cannot ship time-zone support may choose posixct_mode="utc_naive"; the underlying UTC instant is retained and only the display-zone metadata is removed.
• NaN/R missing timestamps become null. Infinite or out-of-range timestamps raise by default; invalid_timestamp="null" is an explicit coercion.
• Homogeneous list-columns use Arrow inference. Heterogeneous values raise by default. list_column_mode="json" produces deterministic UTF-8 JSON, including tagged encodings for binary and non-finite values; "string" is available only when human-readable representation is preferred over structure.
• A factor with an explicit NA level (addNA()) maps values at that level to missing: pandas/Arrow dictionaries cannot hold a null category, and mapping it to "" would collide with a genuine empty-string level.
• Ordered factors (factor(x, ordered = TRUE)) become an ordered pandas.Categorical; plain factors are unordered. to_parquet() marks the underlying Arrow dictionary as ordered too, though the DuckDB staging step currently re-materializes any dictionary-encoded column as a plain string in the final Parquet file (true for ordinary factors as well, not specific to ordering) -- the values are correct, the categorical/dictionary typing itself does not yet survive that step.
• difftime columns become a pandas timedelta64 Series (Arrow duration("us") in Parquet), converted according to the R units attribute (secs, mins, hours, days, or weeks).
• A data.frame column with a dim attribute (an R matrix or array stored as one column) is explicitly unsupported and raises UnsupportedRDS rather than being silently reshaped or misread.

Known limitations

• Rle/other Bioconductor S4 types, sf geometry columns, and data.frame/matrix-valued columns are not specifically recognized; they raise a clear UnsupportedRDS/InvalidRDS, never a silently wrong result.
• POSIXlt values are reconstructed from their wall-clock components (year/mon/mday/hour/min/sec) as timezone-naive timestamps; the zone/gmtoff components are not mapped onto pandas time zones because R zone abbreviations do not reliably resolve to IANA names.
• ALTREP support covers what R itself serializes with custom state: compact integer/real sequences, deferred as.character strings, and the wrap_* wrapper classes (including their attribute slot, so a sorted factor keeps its levels). Third-party ALTREP classes with their own serialization still raise UnsupportedRDS.
• The "native" (non-XDR) binary RDS format has no self-describing byte order. rdsframe validates the header under the reading machine's order and retries the opposite order before giving up, so a cross-endian file produces a clear error or a correct read -- but XDR (saveRDS's default) remains the only portable choice.
• A CHARSXP with no explicit UTF-8/ASCII/latin-1 flag uses the encoding the RDS version-3 header itself declares when it is recognized, else UTF-8. For older files where that guess is wrong, pass encoding="windows-1252" (or any Python codec) to the read functions or --encoding on the CLI. The same override also guards against files that claim UTF-8 in their flags but contain differently-encoded bytes: when encoding= is given, flagged-UTF-8 strings are validated and re-decoded with the override on failure. Without it, flags are trusted as-is -- that zero-validation fast path is deliberate.
• Parquet has no complex-number type, so complex columns are written as STRUCT(real DOUBLE, imag DOUBLE): a lossless representation, not a native semantic type. Consumers must reassemble real + imag*i themselves.
• difftime values are normalized to Arrow duration("us") in Parquet. The elapsed time is exact, but the original display unit ("weeks" vs "days" vs "secs") is not distinguishable from the Parquet type alone.
• Heterogeneous list-columns raise UnsupportedRDS unless list_column_mode="json" or "string" is passed. This is deliberate: silently coercing mixed structures would violate the no-implicit-loss contract, so the caller must pick the representation.
• Environments are read as a plain dict of their contents; the parent scope (enclosure) and lock flag are not represented. S4 objects are read as a dict of their slots with the class recorded under "$r_class".

License

MIT © 2026 Miguel Antonio Manay López.