loadfile

Load tabular data from any storage backend (Google Cloud Storage, AWS S3, Azure, local disk, …) and any common format into a pandas DataFrame with a single call.

from loadfile import load

df = load("gs://my-bucket/data.parquet")
df = load("data/local.csv")
df = load("s3://bucket/export.json", storage_options={"anon": True})

The storage backend is chosen automatically from the URL prefix (powered by fsspec). The file format is detected from the extension (or forced with format=).

The function is also exported as load_data (an alias of load) for compatibility with the original helper.

Install

pip install loadfile            # core: pandas + fsspec + pyarrow (local files)
pip install "loadfile[gcs]"     # Google Cloud Storage (gcsfs)
pip install "loadfile[s3]"      # AWS S3 (s3fs)
pip install "loadfile[azure]"   # Azure (adlfs)
pip install "loadfile[excel]"   # .xlsx support (openpyxl)
pip install "loadfile[calamine]"# fast Rust Excel engine for fast=True
pip install "loadfile[all]"     # everything

Supported formats

Usage

Faster reads with fast=True

df = load("big.csv")              # default: plain pandas (NumPy dtypes)
df = load("big.csv", fast=True)   # Arrow-accelerated: much faster, less memory

fast=True delegates to pyarrow's multi-threaded C++ parser for CSV and JSON-lines and returns Arrow-backed columns (dtype_backend="pyarrow") for every format — typically several times faster with a fraction of the memory, and no slow object dtype. The default stays plain pandas for backwards compatibility. Any explicit reader kwargs you pass override the fast defaults.

Installing loadfile[calamine] (or [all]) additionally enables the fast Rust calamine engine for Excel under fast=True.

Format override and reader options

Any extra keyword arguments are forwarded to the underlying pandas reader:

load("export.tsv", format="csv", sep="\t")
load("book.xlsx", sheet_name="Sheet2")
load("big.csv", usecols=["id", "value"])

Credentials

Pass backend options via storage_options (forwarded to fsspec):

load("gs://bucket/data.parquet", storage_options={"token": "/path/key.json"})
load("s3://bucket/data.csv", storage_options={"key": "...", "secret": "..."})

ZIP archives

A .zip may contain any mix of supported formats. Each member is loaded with the reader matching its own extension.

# Single data file in the zip -> a DataFrame
df = load("archive.zip")

# Multiple data files -> a dict {member_name: DataFrame}
frames = load("archive.zip")               # {"a.csv": ..., "b.parquet": ...}

# Pick one member -> a DataFrame
df = load("archive.zip", filename="a.csv")

# Pick a subset -> a dict of just those members
frames = load("archive.zip", filename=["a.csv", "b.parquet"])

Non-data members (e.g. readme.txt) and nested zips are ignored. A zip with no supported files raises EmptyArchiveError; a missing requested member raises MemberNotFoundError.

Development

pip install -e ".[dev,all]"
pytest

License

MIT