PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
Dependency
- HDFS client and libhdfs for HDFS access
- CPython >= 3.10
Installation and Document build
Installation
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
Documentation
$ cd pfio/docs
$ uv run --extra doc make html
$ open build/html/index.html
How to use
Please refer to the official document for more information about the usage.
Release
Check the official document for latest release procedure.
Run tests locally:
$ uv sync
$ uv run tox --skip-env doc
Bump version numbers in pfio/version.py .
Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:
$ git tag -s X.Y.Z
$ git push --tags
Build:
$ rm -rf dist
$ uv build
Release to PyPI:
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
5 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pfio-2.10.0.tar.gz.
File metadata
- Download URL: pfio-2.10.0.tar.gz
- Upload date:
- Size: 41.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5e240350b86066cb63b7b8dc7062c16c26c0f30761c3f1bb63d49325f0218c2d
|
|
| MD5 |
9f344d26ec34f8c7bb61f29c6d27883b
|
|
| BLAKE2b-256 |
f12ef582f5384dda77f37d7110fb0e7c9043b749f1ee67becde92ef2a51ce01e
|
File details
Details for the file pfio-2.10.0-py3-none-any.whl.
File metadata
- Download URL: pfio-2.10.0-py3-none-any.whl
- Upload date:
- Size: 49.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0713390bdb5c9b08b45a8e74a1d47cb7958598ed134cc3cd04d2c3ab420ecfc2
|
|
| MD5 |
3e27da9495e224ac5ba0b82e772a3f5c
|
|
| BLAKE2b-256 |
848d6221c8d7c2e98ae4aa67fbba123d6b19db3cff3ab6f01c04a1227f22d699
|
