rdsframe 0.4.0a7

Fast, memory-conscious reader for RDS data.frames

rdsframe

rdsframe reads common R data.frame and data.table objects from binary RDS files directly into pandas. Numeric vectors are filled in their final NumPy allocation, which avoids the large intermediate Python object tree created by general-purpose R serialization readers.

Status: 0.4.0a7 pre-release. Validate results against R for critical pipelines. Unsupported R structures fail explicitly; they are never silently coerced.

Features

• R serialization versions 2 and 3, XDR and native binary formats.
• Uncompressed, gzip, bzip2, and xz containers.
• Integer, double, logical, character, factor (ordered and unordered), Date, POSIXct, POSIXlt, difftime, raw, and the ALTREP representations R actually serializes (compact integer/real sequences, deferred as.character strings, sort()/attribute wrappers). Complex columns become complex pandas series or STRUCT(real, imag) in Parquet.
• Batched string parsing: STRSXP columns are decoded (or structurally skipped) from large chunks instead of three stream reads per element, which is what makes text-heavy catalogs practical (see BENCHMARKS.md).
• Explicit policies for time zones, invalid timestamps, and heterogeneous list-columns; lossy conversion is never activated implicitly.
• A single data.frame or a named list of data.frame objects -- and, via read_r_object() or rdsframe dump, any other supported R value: plain lists, nested structures, matrices, S4 objects (as a dict of slots), and environments (as a dict of their contents).
• Reads from a path, raw bytes, or any seekable binary stream.
• Character row names become the pandas index; R's compact default row numbering stays a RangeIndex.
• Nullable pandas integer/boolean dtypes and categorical factors.
• Column-staged Parquet export through DuckDB and a command-line interface.
• Configurable defensive limits for untrusted or unexpectedly large inputs.
• Low-allocation table catalogs and selective extraction by index, name, or (for read_rds()) column, plus materialize_uncompressed() for repeated seek-based access to large compressed files.
• Validated against a golden-file corpus written by R 4.5.0 itself (tests/data/r450, generated by tests/data/gen_fixtures.R).

RData workspaces, ASCII serialization, closures/language objects, and custom third-party ALTREP classes are intentionally unsupported -- and fail with an error that names the R type, so applications can tell users exactly why a slower general-purpose fallback is being used.

Install

pip install rdsframe

For Parquet export:

pip install "rdsframe[parquet]"

Python API

from rdsframe import read_rds

data = read_rds("measurements.rds")
print(data.head())

A named R list returns dict[str, pandas.DataFrame]. To require the legacy first-table behavior, use read_rds_dataframe().

For text-heavy data that still needs to become a pandas DataFrame, keep strings in Arrow-backed pandas arrays:

data = read_rds("measurements.rds", strings="pyarrow")

To read only a few fields from a wide single data.frame, select columns by zero-based index or exact name. Unselected columns are structurally skipped: they are never allocated as NumPy arrays or pandas objects.

data = read_rds("measurements.rds", columns=["station", "value", "date"])

columns requires the RDS root to be a single data.frame. For a multi-table RDS, select the table first with extract_rds_tables() or to_parquet(). Selecting by name performs one extra structural pass to resolve names to positions (bounded memory, no payload allocation); integer indices skip that pass entirely.

The saving depends on the type of the skipped columns, not just their count. RDS stores columns sequentially with variable-length encoding: skipping a numeric, logical, or raw column in an uncompressed file is a real seek() (near-zero cost); skipping a character column still visits every row to find where it ends, the same traversal a full read does, so it mainly saves the decode/interning/pandas-object cost rather than the row scan itself. columns is most valuable for RAM (skipped columns are never allocated) and for wide tables with many unwanted numeric fields.

Reading non-tabular RDS files

Not every RDS file is a data.frame. read_rds() raises UnsupportedRDS for those on purpose, since silently coercing a plain list into a table would be worse than failing loudly. Use read_r_object() instead:

from rdsframe import read_r_object

data = read_r_object("workspace.rds")

A named R list becomes a dict; an unnamed list becomes a list; a nested data.frame anywhere inside either still becomes a pandas DataFrame. Atomic vectors get the same type rules as a data.frame column (factor, Date, POSIXct, difftime, matrices via a dim attribute) but come back as a plain Python scalar or list rather than a pandas Series. This is a general, exploratory reader, not the fast path: it materializes the whole structure in memory rather than streaming it, so prefer read_rds() / to_parquet() whenever the file actually is tabular.

from rdsframe import ReaderLimits, read_rds, to_parquet

limits = ReaderLimits(max_vector_length=100_000_000)
data = read_rds("input.rds", limits=limits, strings="string")

tables = to_parquet("input.rds", "output", compression="zstd")

Safety and fidelity policies are explicit:

tables = to_parquet(
    "input.rds",
    "output",
    max_tables=100,            # fail before writing partial results
    max_root_items=10_000,
    posixct_mode="preserve",   # or "utc_naive" for constrained executables
    invalid_timestamp="error", # or "null" to opt into coercion
    list_column_mode="infer",  # or explicit "json" / "string"
)

max_tables never truncates a file. Exceeding a configured limit raises RDSLimitError, cleans temporary data, and leaves no apparently successful partial output.

The staging policy is adaptive. By default, a temporary Parquet contains at most 16 columns or approximately 128 MiB of Arrow buffers, whichever is reached first. This reduces temporary-file count on wide tables while bounding memory:

tables = to_parquet(
    "input.rds",
    "output",
    stage_max_columns=16,
    stage_max_bytes=128 * 1024 * 1024,
    gc_collect_every=16,  # use 0 to disable explicit cyclic GC
)

CLI

rdsframe inspect input.rds
rdsframe list input.rds --catalog input.rdsframe.json
rdsframe convert input.rds output/ \
  --table-name measurements \
  --catalog input.rdsframe.json \
  --max-tables 100 \
  --list-column-mode json \
  --invalid-timestamp null
rdsframe dump input.rds

Run rdsframe convert --help for staging, memory, time-zone and limit controls.

rdsframe dump prints any supported R object -- not only data.frames -- as an indented tree, so an unknown RDS file can be explored before deciding how to convert it. Deeply nested structures are summarized rather than exploded: --max-items bounds children per node (default 10), --max-depth bounds nesting (default 8), and --json emits the complete object as JSON instead (data.frames appear as {"$r_type": "data.frame", ...} objects).

$ rdsframe dump results.rds --max-items 2 --max-depth 3
<list, 10 items>
  [0] <list, 10 items>
    [0] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    [1] <data.frame 20000 rows x 4 cols> [n:float64, c:float64, auc:float64, rc:float64]
    ... (+8 more items)
  [1] <list, 10 items>
    ...

List and selectively extract tables

Discover tables without building their pandas, NumPy, Arrow, or Parquet payloads:

from rdsframe import list_rds_tables

catalog = list_rds_tables("workspace.rds")
for table in catalog.tables:
    print(table.index, table.name, table.rows, table.columns)

catalog.save("workspace.rdsframe.json")

Extract by zero-based index in one pass:

from rdsframe import extract_rds_tables

extract_rds_tables("workspace.rds", "output", [0, 2, 7])

Or select by name while reusing the validated catalog:

from rdsframe import RDSCatalog, extract_rds_tables

catalog = RDSCatalog.load("workspace.rdsframe.json")
extract_rds_tables(
    "workspace.rds",
    "output",
    ["measurements", "stations"],
    catalog=catalog,
)

The catalog stores the absolute path, file size and nanosecond modification time. A stale or foreign catalog raises RDSCatalogError before conversion.

RDS is sequential and normally stores list names after all elements. Therefore:

• listing avoids column allocations and temporary outputs, but compressed input must still be traversed and decompressed;
• uncompressed binary RDS payloads are skipped with direct seeks; compressed payloads are discarded through a reusable bounded buffer;
• integer selection without a catalog converts only chosen tables but may scan the remaining stream to recover original names;
• a validated catalog enables early stop after the last selected table and is the fastest repeated workflow;
• name selection without a catalog performs one structural listing pass and a second selective conversion pass.

The source distribution includes BENCHMARKS.md with preliminary reproducible results and their limitations.

Memory model

read_rds() returns pandas objects and therefore requires the resulting table to fit in memory. Numeric columns avoid a full-size temporary bytes buffer.

to_parquet() is the large-file path: it constructs Arrow-native columns and stages memory-bounded groups of columns before DuckDB combines them with a positional join. Character vectors are built from offsets, UTF-8 data and validity buffers without retaining a Python str object for every row. Peak parser memory is tied to the current column plus the configured staging batch, not to the entire data frame. DuckDB can spill merge work to disk, controlled by memory_limit and temp_directory.

For the lowest possible peak, set stage_max_columns=1. For faster conversion of wide, narrow-column tables, increase the batch limits after benchmarking.

The result is directly queryable without loading it in memory:

SELECT category, avg(value)
FROM read_parquet('output/measurements.parquet')
WHERE date >= DATE '2025-01-01'
GROUP BY category;

Development

python -m pip install -e ".[dev,parquet]"
pytest
ruff check .
python -m build
twine check dist/*

Before a public release, run the test matrix and publish to TestPyPI first. The project lives at https://github.com/mmanaylopez/rdsframe.

RData and fallback integration

rdsframe deliberately rejects .RData/.rda workspaces. Applications that need both formats should inspect the container and use rdsframe for supported binary RDS files, retaining rdata or another general reader as fallback for RData, unsupported ALTREP classes, and richer R objects. An UnsupportedRDS exception is the explicit signal to activate that fallback.

Conversion policies

• POSIXct preserves its R time-zone attribute by default. Applications that cannot ship time-zone support may choose posixct_mode="utc_naive"; the underlying UTC instant is retained and only the display-zone metadata is removed.
• NaN/R missing timestamps become null. Infinite or out-of-range timestamps raise by default; invalid_timestamp="null" is an explicit coercion.
• Homogeneous list-columns use Arrow inference. Heterogeneous values raise by default. list_column_mode="json" produces deterministic UTF-8 JSON, including tagged encodings for binary and non-finite values; "string" is available only when human-readable representation is preferred over structure.
• Ordered factors (factor(x, ordered = TRUE)) become an ordered pandas.Categorical; plain factors are unordered. to_parquet() marks the underlying Arrow dictionary as ordered too, though the DuckDB staging step currently re-materializes any dictionary-encoded column as a plain string in the final Parquet file (true for ordinary factors as well, not specific to ordering) -- the values are correct, the categorical/dictionary typing itself does not yet survive that step.
• difftime columns become a pandas timedelta64 Series (Arrow duration("us") in Parquet), converted according to the R units attribute (secs, mins, hours, days, or weeks).
• A data.frame column with a dim attribute (an R matrix or array stored as one column) is explicitly unsupported and raises UnsupportedRDS rather than being silently reshaped or misread.

Known limitations

• Rle/other Bioconductor S4 types, sf geometry columns, and data.frame/matrix-valued columns are not specifically recognized; they raise a clear UnsupportedRDS/InvalidRDS, never a silently wrong result.
• POSIXlt values are reconstructed from their wall-clock components (year/mon/mday/hour/min/sec) as timezone-naive timestamps; the zone/gmtoff components are not mapped onto pandas time zones because R zone abbreviations do not reliably resolve to IANA names.
• ALTREP support covers what R itself serializes with custom state: compact integer/real sequences, deferred as.character strings, and the wrap_* wrapper classes (including their attribute slot, so a sorted factor keeps its levels). Third-party ALTREP classes with their own serialization still raise UnsupportedRDS.
• The "native" (non-XDR) binary RDS format has no self-describing byte order. rdsframe validates the header under the reading machine's order and retries the opposite order before giving up, so a cross-endian file produces a clear error or a correct read -- but XDR (saveRDS's default) remains the only portable choice.
• A CHARSXP with no explicit UTF-8/ASCII/latin-1 flag uses the encoding the RDS version-3 header itself declares when it is recognized, else UTF-8. For older files where that guess is wrong, pass encoding="windows-1252" (or any Python codec) to the read functions or --encoding on the CLI. The same override also guards against files that claim UTF-8 in their flags but contain differently-encoded bytes: when encoding= is given, flagged-UTF-8 strings are validated and re-decoded with the override on failure. Without it, flags are trusted as-is -- that zero-validation fast path is deliberate.
• Parquet has no complex-number type, so complex columns are written as STRUCT(real DOUBLE, imag DOUBLE): a lossless representation, not a native semantic type. Consumers must reassemble real + imag*i themselves.
• difftime values are normalized to Arrow duration("us") in Parquet. The elapsed time is exact, but the original display unit ("weeks" vs "days" vs "secs") is not distinguishable from the Parquet type alone.
• Heterogeneous list-columns raise UnsupportedRDS unless list_column_mode="json" or "string" is passed. This is deliberate: silently coercing mixed structures would violate the no-implicit-loss contract, so the caller must pick the representation.
• Environments are read as a plain dict of their contents; the parent scope (enclosure) and lock flag are not represented. S4 objects are read as a dict of their slots with the class recorded under "$r_class".

License

MIT © 2026 Miguel Antonio Manay López.