PFN IO library
Project description
PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
Dependency
- HDFS client and libhdfs for HDFS access
- CPython >= 3.9
Installation and Document build
Installation
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
Documentation
$ cd pfio/docs
$ make html
$ open build/html/index.html
How to use
Please refer to the official document for more information about the usage.
Release
Check the official document for latest release procedure.
Run tests locally:
$ pip install tox
$ tox
Bump version numbers in pfio/version.py
.
Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:
$ git tag -s X.Y.Z
$ git push --tags
Build:
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
Release to PyPI:
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pfio-2.9.0.tar.gz
(41.2 kB
view details)
Built Distribution
pfio-2.9.0-py3-none-any.whl
(48.5 kB
view details)
File details
Details for the file pfio-2.9.0.tar.gz
.
File metadata
- Download URL: pfio-2.9.0.tar.gz
- Upload date:
- Size: 41.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a23d3bbc7baf85cad6cdb22e1582ad59a786baef96bbaac699edc115080ab70 |
|
MD5 | 98147f49d6d01fe95a24b8fdfd3728b7 |
|
BLAKE2b-256 | 5120d9a873a2b552baff2b2983a7ed67c506bfcfad45f178a2b593fdeb5bd568 |
File details
Details for the file pfio-2.9.0-py3-none-any.whl
.
File metadata
- Download URL: pfio-2.9.0-py3-none-any.whl
- Upload date:
- Size: 48.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41e6319ae51df5a58eef217896fc941f728328274a9233b5b645cfc62114b02d |
|
MD5 | 76b5a1d48826406e7ee5eb118e2373a0 |
|
BLAKE2b-256 | fd0f18a64949c3e8bb2e309c1d2f4670e2d23aa9fbd1a9e08eaf224725a4429f |