dlt-filesystem

Filesystem and blob storage sources and destinations for dlt.

dlt ships its own filesystem source, and this package is the delta over it:

16 readers behind 19 routed format keys, against dlt's four. CSV (pyarrow, DuckDB and a headerless variant), JSON and JSONL, Parquet, ORC, Avro, Feather, BSON, spreadsheets (xlsx and ods), XML, YAML, MessagePack and CBOR.
A filesystem resource that refuses to load nothing. A concrete selection matching no file raises rather than returning an empty table; globs stay empty-safe.
A modification-date resolver that covers the schemes dlt's table does not, including r2, oss, hdfs, smb, ftp and webdav, and a pyarrow-backed client addressed as s3://.
An Arrow readinto shim, without which reading a .gz fails wherever isal is importable.

The entry points are a superset of dlt's own, so a pipeline already on dlt.sources.filesystem can move across without changing its call.

Install

pip install dlt-filesystem

The long-tail formats (XML, YAML, MessagePack, CBOR) carry their decoders in an extra:

pip install 'dlt-filesystem[iterable]'

Usage

import dlt
from dlt_filesystem.source.adapter import readers

pipeline = dlt.pipeline(destination="duckdb", dataset_name="inbox")
pipeline.run(
    readers(bucket_url="s3://bucket/prefix", file_glob="*.parquet").read_parquet()
)

Documentation

https://omniload.readthedocs.io/supported-sources/filesystem.html

License

MIT. See LICENSE.

Metadata

Release files for dlt-filesystem 0.18.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dlt-filesystem 0.18.0
File	Size	Uploaded
dlt_filesystem-0.18.0.tar.gz	150.1 kB	Sep 23, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for dlt-filesystem 0.18.0
File	Interpreter	ABI	Platform	Reset
dlt_filesystem-0.18.0-py3-none-any.whl 116.7 kB Sep 23, 2026	Python 3	none	any	Details

Total release size: 266.7 kB

Release files / dlt_filesystem-0.18.0.tar.gz

Download URL	dlt_filesystem-0.18.0.tar.gz
Size	150.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`25e04cc4b8e607f303765ad499dd5ba5195ebccec3173f988e480b4fe30cba1e`
BLAKE2b-256 checksum How to use checksums	`95ee851302d8f3d73da194439ee0e74757da47cbcd65d568d5ac0e308031249d`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.11`

Release files / dlt_filesystem-0.18.0-py3-none-any.whl

Download URL	dlt_filesystem-0.18.0-py3-none-any.whl
Size	116.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`911439161ab832d8c93a1f2afc97ae52320478ec752b6436cc1620954b0f113b`
BLAKE2b-256 checksum How to use checksums	`336d3931e8e33b172d9b3d01ed159a12a614ce596b35a7593ef14ea75b5c6dc1`
Upload date	Sep 23, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.11`