Skip to main content

PFN IO library

Project description

PFIO

Doc Status Badge

PFIO is an IO abstraction library developed by PFN, optimized for deep learning training with batteries included. It supports

  • Filesystem API abstraction with unified error semantics,
  • Explicit user-land caching system,
  • IO performance tracing and metrics stats, and
  • Fileset container utilities to save metadata.

Dependency

  • HDFS client and libhdfs for HDFS access
  • CPython >= 3.9

Installation and Document build

Installation

$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .

Documentation

$ cd pfio/docs
$ make html
$ open build/html/index.html

How to use

Please refer to the official document for more information about the usage.

Release

Check the official document for latest release procedure.

Run tests locally:

$ pip install tox
$ tox

Bump version numbers in pfio/version.py .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged, tag a release:

$ git tag -s X.Y.Z
$ git push --tags

Build:

$ rm -rf dist
$ pip3 install --user build
$ python3 -m build

Release to PyPI:

$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pfio-2.9.0.tar.gz (41.2 kB view details)

Uploaded Source

Built Distribution

pfio-2.9.0-py3-none-any.whl (48.5 kB view details)

Uploaded Python 3

File details

Details for the file pfio-2.9.0.tar.gz.

File metadata

  • Download URL: pfio-2.9.0.tar.gz
  • Upload date:
  • Size: 41.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for pfio-2.9.0.tar.gz
Algorithm Hash digest
SHA256 0a23d3bbc7baf85cad6cdb22e1582ad59a786baef96bbaac699edc115080ab70
MD5 98147f49d6d01fe95a24b8fdfd3728b7
BLAKE2b-256 5120d9a873a2b552baff2b2983a7ed67c506bfcfad45f178a2b593fdeb5bd568

See more details on using hashes here.

File details

Details for the file pfio-2.9.0-py3-none-any.whl.

File metadata

  • Download URL: pfio-2.9.0-py3-none-any.whl
  • Upload date:
  • Size: 48.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.12

File hashes

Hashes for pfio-2.9.0-py3-none-any.whl
Algorithm Hash digest
SHA256 41e6319ae51df5a58eef217896fc941f728328274a9233b5b645cfc62114b02d
MD5 76b5a1d48826406e7ee5eb118e2373a0
BLAKE2b-256 fd0f18a64949c3e8bb2e309c1d2f4670e2d23aa9fbd1a9e08eaf224725a4429f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page