Skip to main content

Read GCS and local paths with the same interface, clone of tensorflow.io.gfile

Project description

blobfile

This is a standalone clone of TensorFlow's gfile, supporting both local paths and gs:// paths.

Writing to a remote path will not actually perform the write incrementally, so don't write to a log file this way. By default reads and writes are streamed, set streaming=False to BlobFile to do a single copy operation per file instead.

The main function is BlobFile, a replacement for GFile. There are also a few additional functions, basename, dirname, and join, which mostly do the same thing as their os.path namesakes, only they also support gs:// paths. There are also a few extra functions:

  • cache_key - returns a cache key that can be used for the path (this is not guaranteed to change when the content changes, but should hopefully do that)
  • retry - useful for retrying since network operations often fail
  • get_url - returns a url for a path
  • md5 - get the md5 hash for a path, for GCS this is fast, but for other backends this may be slow

A number of existing gfile functions are currently not implemented.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for blobfile, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size blobfile-0.2.0-py3-none-any.whl (11.7 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page