Skip to main content

PFN IO library

Project description


PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

  • Filesystem API abstraction with unified error semantics,
  • Explicit user-land caching system,
  • IO performance tracing and metrics stats, and
  • Fileset container utilities to save metadata.


  • HDFS client and libhdfs for HDFS access
  • CPython >= 3.6

Installation and Document build


$ git clone
$ cd pfio
$ pip install .


$ cd pfio/docs
$ make html
$ open build/html/index.html


$ cd pfio
$ pip install .[test]
$ pytest tests/

How to use

Please refer to the official document for more information about the usage.


Check the official document for latest release procedure.

Run tests locally:

$ pip install --user -e .[test]
$ pytest

Bump version numbers in pfio/ .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:

$ git tag -s X.Y.Z
$ git push --tags


$ rm -rf dist
$ pip3 install --user build
$ python3 -m build

Release to PyPI:

$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pfio-2.4.0.tar.gz (34.7 kB view hashes)

Uploaded source

Built Distribution

pfio-2.4.0-py3-none-any.whl (42.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page