dlt-filesystem
Filesystem and blob storage sources and destinations for dlt.
dlt ships its own filesystem source, and this package is the delta over it:
- 16 readers behind 19 routed format keys, against dlt's four. CSV (pyarrow,
DuckDB and a headerless variant), JSON and JSONL, Parquet, ORC, Avro, Feather,
BSON, spreadsheets (
xlsxandods), XML, YAML, MessagePack and CBOR. - A
filesystemresource that refuses to load nothing. A concrete selection matching no file raises rather than returning an empty table; globs stay empty-safe. - A modification-date resolver that covers the schemes dlt's table does not,
including
r2,oss,hdfs,smb,ftpandwebdav, and a pyarrow-backed client addressed ass3://. - An Arrow
readintoshim, without which reading a.gzfails whereverisalis importable.
The entry points are a superset of dlt's own, so a pipeline already on
dlt.sources.filesystem can move across without changing its call.
Install
pip install dlt-filesystem
The long-tail formats (XML, YAML, MessagePack, CBOR) carry their decoders in an extra:
pip install 'dlt-filesystem[iterable]'
Formats whose decoder is a sizeable or single-purpose package carry it in an extra of their own. Without it, the format still routes and its reader raises an error naming the extra to install.
| Extra | Formats |
|---|---|
bson |
BSON |
duckdb |
CSV read with DuckDB (csv_duckdb) |
spreadsheet |
XLSX (read and write), ODS |
vortex |
Vortex (Python 3.11+) |
pip install 'dlt-filesystem[spreadsheet]'
Usage
import dlt
from dlt_filesystem.source.adapter import readers
pipeline = dlt.pipeline(destination="duckdb", dataset_name="inbox")
pipeline.run(
readers(bucket_url="s3://bucket/prefix", file_glob="*.parquet").read_parquet()
)
Documentation
https://omniload.readthedocs.io/supported-sources/filesystem.html
License
MIT. See LICENSE.
Release files for dlt-filesystem 0.20.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dlt_filesystem-0.20.0.tar.gz | 162.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dlt_filesystem-0.20.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 285.4 kB
Release files / dlt_filesystem-0.20.0.tar.gz
| Download URL | dlt_filesystem-0.20.0.tar.gz |
|---|---|
| Size | 162.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1095f25953510911d72700038c44a189104ec9344b92ed732ba9d10b91221317
|
|
BLAKE2b-256 checksum How to use checksums |
a1f8e8405956ba5c352f2b15132f6e26f2c636729eab04995ed471db069d38d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency logRelease files / dlt_filesystem-0.20.0-py3-none-any.whl
| Download URL | dlt_filesystem-0.20.0-py3-none-any.whl |
|---|---|
| Size | 123.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0c0d58bbfeedccad9b1e1d09f0b94d55b061fcb232208bdb8a28b39f722cc8e3
|
|
BLAKE2b-256 checksum How to use checksums |
1d465282a574000b1b4ffff5738ccadd21c0cd0eab0f2297c7b1a70452116524
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.
Transparency log
