Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.23.1 2.7.x Dec 15, 2021
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 25719911b974acc422e2fa4e84bf394c68dc3ff78818a9c469bb7e37843fd8ec
MD5 28373c5634afbd4631fd40ef3ea01382
BLAKE2b-256 360a6aee7102a61265930b74901f1a1d6a1048f3d1fde80ddabb5f7c7b91fa15

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 af26466008c93fd69159affacd9d821fca111e3a51bd76f025ca2f377dcc255f
MD5 8054dca207cf0ed0b2c1d8d57d36e2df
BLAKE2b-256 35db6bd0d4a0f800cf356e3468b975b0934fec48f27cf73abb403ecfa7457383

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 23af959fb227859f6a1be5be5604c6baba1e4cf619f91cb7b184b4d7cc7106a6
MD5 58d1991d8e57052cb7f3a3ef2483a963
BLAKE2b-256 0cda4922aaf4347ec7d0144033ed8d4338970f6514eb0ae95cf9772fdb05fdfa

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 0d5e19e99f074efb5059fd3133e60a99eb79d0e53ef707154fe09f9cff613acd
MD5 659b2ec181afc309f4f7b7b58ea3a704
BLAKE2b-256 1653a1bee5366db4c5e854374e33711b5b4ab696a7f5b164e7848f0722bdb465

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 4029c732419f1a939243a1d90511ded85f1574b5f470a7ca58e4c13cfe0fa13d
MD5 090db54368feb7e1ba09bc3a4500ba3c
BLAKE2b-256 1ab3462bc414f1a961b24a4a82299d8f46f74a42e91e4b49bcf24b43ab9f6d99

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0b02b894d8bc917b34f972c55d8b425f00b9b41c0560311e7a16fe4b0387441d
MD5 482b5ff32d3821e641bdef75d6d1e210
BLAKE2b-256 c3975e8d3f38847c07bcf0085398c73475f954c62241798074529b09cf63441a

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 505e5c1c1cdaa69c61051319b46fd8a00e0f4025e6c1c17f7f9931cfe992347a
MD5 7a7b08966c0b96932845e22234bdb277
BLAKE2b-256 7944dc2ca512dd370b327a354073560959197dc58a3869cf22041c0717e9e73d

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e655057514092c5f789d846f23244f99922d3cf7fba7fe91380950a3dd9d6505
MD5 7e557530eec1a0a2b5b8ce54b6b2832e
BLAKE2b-256 ba5810a6aaff0904b910a52d04b9c3d7e4ac4bc0fa9a6aafedd0bccc1d1760b3

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 70430b24f59e02dc28e651e93ca3d7752b2e5a3660d614cd6da18265ae532156
MD5 4256d6fb4581ba96825d14ea76f1987f
BLAKE2b-256 14b9cd8f534350449bdd70b791af3f1f0c966c224f015096af75dde6875adeee

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5b7a3bbcbe1697b3905a6a005dc2bc968c53f35def4d438f454e16b0e2396fcd
MD5 c5a487a3f864b08d383750bcf352b222
BLAKE2b-256 34fec0d47a029166e98453880ce2245ffc30383c21e4be74b97eeaf17ab1a822

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 ac89e80ab6c2ef123240cf07a991f07d50166b3a888efe23b2177cc737029e3a
MD5 16cc880454c68d53eabd7eecf9c3913d
BLAKE2b-256 360a1327e5392097f0984d6d71e1e33e6cea5b0be5494a861559ca0d0378f752

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.1.dev20211216204107-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6014d244dd1f09b18129a08993a71d49405057a709e3ce2cfd32f05a6a48c68e
MD5 4ac9951ef52a15a6bc3f5f67bbd2d467
BLAKE2b-256 9e9862475d0dc309380576d0d96ff3670cdc0292ffef43dc4b11922ca546e67a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page