Skip to main content

TensorFlow IO

Project description




TensorFlow I/O

GitHub CI PyPI License Documentation

TensorFlow I/O is a collection of file systems and file formats that are not available in TensorFlow's built-in support. A full list of supported file systems and file formats by TensorFlow I/O can be found here.

The use of tensorflow-io is straightforward with keras. Below is an example to Get Started with TensorFlow with the data processing aspect replaced by tensorflow-io:

import tensorflow as tf
import tensorflow_io as tfio

# Read the MNIST data into the IODataset.
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz",
)

# Shuffle the elements of the dataset.
d_train = d_train.shuffle(buffer_size=1024)

# By default image data is uint8, so convert to float32 using map().
d_train = d_train.map(lambda x, y: (tf.image.convert_image_dtype(x, tf.float32), y))

# prepare batches the data just like any other tf.data.Dataset
d_train = d_train.batch(32)

# Build the model.
model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(512, activation=tf.nn.relu),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation=tf.nn.softmax),
    ]
)

# Compile the model.
model.compile(
    optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)

# Fit the model.
model.fit(d_train, epochs=5, steps_per_epoch=200)

In the above MNIST example, the URL's to access the dataset files are passed directly to the tfio.IODataset.from_mnist API call. This is due to the inherent support that tensorflow-io provides for HTTP/HTTPS file system, thus eliminating the need for downloading and saving datasets on a local directory.

NOTE: Since tensorflow-io is able to detect and uncompress the MNIST dataset automatically if needed, we can pass the URL's for the compressed files (gzip) to the API call as is.

Please check the official documentation for more detailed and interesting usages of the package.

Installation

Python Package

The tensorflow-io Python package can be installed with pip directly using:

$ pip install tensorflow-io

People who are a little more adventurous can also try our nightly binaries:

$ pip install tensorflow-io-nightly

To ensure you have a version of TensorFlow that is compatible with TensorFlow-IO, you can specify the tensorflow extra requirement during install:

pip install tensorflow-io[tensorflow]

Similar extras exist for the tensorflow-gpu, tensorflow-cpu and tensorflow-rocm packages.

Docker Images

In addition to the pip packages, the docker images can be used to quickly get started.

For stable builds:

$ docker pull tfsigio/tfio:latest
$ docker run -it --rm --name tfio-latest tfsigio/tfio:latest

For nightly builds:

$ docker pull tfsigio/tfio:nightly
$ docker run -it --rm --name tfio-nightly tfsigio/tfio:nightly

R Package

Once the tensorflow-io Python package has been successfully installed, you can install the development version of the R package from GitHub via the following:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("tensorflow/io", subdir = "R-package")

TensorFlow Version Compatibility

To ensure compatibility with TensorFlow, it is recommended to install a matching version of TensorFlow I/O according to the table below. You can find the list of releases here.

TensorFlow I/O Version TensorFlow Compatibility Release Date
0.23.0 2.7.x Dec 14, 2021
0.22.0 2.7.x Nov 10, 2021
0.21.0 2.6.x Sep 12, 2021
0.20.0 2.6.x Aug 11, 2021
0.19.1 2.5.x Jul 25, 2021
0.19.0 2.5.x Jun 25, 2021
0.18.0 2.5.x May 13, 2021
0.17.1 2.4.x Apr 16, 2021
0.17.0 2.4.x Dec 14, 2020
0.16.0 2.3.x Oct 23, 2020
0.15.0 2.3.x Aug 03, 2020
0.14.0 2.2.x Jul 08, 2020
0.13.0 2.2.x May 10, 2020
0.12.0 2.1.x Feb 28, 2020
0.11.0 2.1.x Jan 10, 2020
0.10.0 2.0.x Dec 05, 2019
0.9.1 2.0.x Nov 15, 2019
0.9.0 2.0.x Oct 18, 2019
0.8.1 1.15.x Nov 15, 2019
0.8.0 1.15.x Oct 17, 2019
0.7.2 1.14.x Nov 15, 2019
0.7.1 1.14.x Oct 18, 2019
0.7.0 1.14.x Jul 14, 2019
0.6.0 1.13.x May 29, 2019
0.5.0 1.13.x Apr 12, 2019
0.4.0 1.13.x Mar 01, 2019
0.3.0 1.12.0 Feb 15, 2019
0.2.0 1.12.0 Jan 29, 2019
0.1.0 1.12.0 Dec 16, 2018

Performance Benchmarking

We use github-pages to document the results of API performance benchmarks. The benchmark job is triggered on every commit to master branch and facilitates tracking performance w.r.t commits.

Contributing

Tensorflow I/O is a community led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see:

Build Status and CI

Build Status
Linux CPU Python 2 Status
Linux CPU Python 3 Status
Linux GPU Python 2 Status
Linux GPU Python 3 Status

Because of manylinux2010 requirement, TensorFlow I/O is built with Ubuntu:16.04 + Developer Toolset 7 (GCC 7.3) on Linux. Configuration with Ubuntu 16.04 with Developer Toolset 7 is not exactly straightforward. If the system have docker installed, then the following command will automatically build manylinux2010 compatible whl package:

#!/usr/bin/env bash

ls dist/*
for f in dist/*.whl; do
  docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*

It takes some time to build, but once complete, there will be python 3.5, 3.6, 3.7 compatible whl packages available in wheelhouse directory.

On macOS, the same command could be used. However, the script expects python in shell and will only generate a whl package that matches the version of python in shell. If you want to build a whl package for a specific python then you have to alias this version of python to python in shell. See .github/workflows/build.yml Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration. GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test. Again, because of the manylinux2010 requirement, on Linux whl packages are always built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems with different python3 versions to ensure a good coverage:

Python Ubuntu 18.04 Ubuntu 20.04 macOS + osx9 Windows-2019
2.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: N/A
3.7 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:

TensorFlow I/O has integrations with many systems and cloud vendors such as Prometheus, Apache Kafka, Apache Ignite, Google Cloud PubSub, AWS Kinesis, Microsoft Azure Storage, Alibaba Cloud OSS etc.

We tried our best to test against those systems in our continuous integration whenever possible. Some tests such as Prometheus, Kafka, and Ignite are done with live systems, meaning we install Prometheus/Kafka/Ignite on CI machine before the test is run. Some tests such as Kinesis, PubSub, and Azure Storage are done through official or non-official emulators. Offline tests are also performed whenever possible, though systems covered through offine tests may not have the same level of coverage as live systems or emulators.

Live System Emulator CI Integration Offline
Apache Kafka :heavy_check_mark: :heavy_check_mark:
Apache Ignite :heavy_check_mark: :heavy_check_mark:
Prometheus :heavy_check_mark: :heavy_check_mark:
Google PubSub :heavy_check_mark: :heavy_check_mark:
Azure Storage :heavy_check_mark: :heavy_check_mark:
AWS Kinesis :heavy_check_mark: :heavy_check_mark:
Alibaba Cloud OSS :heavy_check_mark:
Google BigTable/BigQuery to be added
Elasticsearch (experimental) :heavy_check_mark: :heavy_check_mark:
MongoDB (experimental) :heavy_check_mark: :heavy_check_mark:

References for emulators:

Community

Additional Information

License

Apache License 2.0

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 0a7f1cb6ae2028b1aa47ad8309a2e06af81bdcd5ed39a5de094cbcfb5e9a423b
MD5 402f743291bd12e6344f9f853c507154
BLAKE2b-256 24a56e27a5684c004e422d9c41eca2daa93dd6c8c4d84c7f312bd2e3b3f0116a

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 506f768b1255394dd6cdd5830b1a3bf83e1ebe70633d8fed6ce4b009f11fbf60
MD5 4b2ac242d7d1f35ed1a5d4704fb3e87d
BLAKE2b-256 c9b7817a697252171be2b67a42217f13afc34eaf01453bdda10cb68472802805

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 e8fc1b1dd640171caf6bd77c22c0431c5cb96fb4370875972643e4f56badb50a
MD5 c6a98436d273927689f7afb74cdf77bb
BLAKE2b-256 39662d7813278a2b3c3567bfc0131c7ec08e42cd1212ca5564d0f42cbdcab9fb

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 220e0ddf80f0cb8ae1216f060d83b5e4d8f1d210e87083b52c853748c1b2b492
MD5 430e8ef21cfa09b3b3ceabbfb1396303
BLAKE2b-256 455e712ac8e40d959e70b2c6980a1b4fe7f3b30a655c7fd045624127296cee49

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e9b98118b92fa3e7dfad54cd32448e351ab3672b62edbf0fc2cf251c429c5cf9
MD5 7a320c6f10ccb218ee53a745951603cf
BLAKE2b-256 7852ac99f545a69ba9ca821810f3ad361bca4eee9da342ff698b8cd197165023

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4664e75ee5d363fe4dd64eebc47a0b5768f21ff3e8a012566428199a1b48d362
MD5 b63c610cb9ea5b8ef8c336f9543817a1
BLAKE2b-256 3ae0860cb7061265775b6a5d14432e8c7e29e6054bc84ccef49e62a6bb4ecec1

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 c4df66ab252e096b09a7c761489df11474d06bc6c1e5d9b9335c00ae6d514ae0
MD5 4122930cfe0c4441371eca52bacf4e8b
BLAKE2b-256 98ef0cb63094bb97ec626073d6ff7e355861edb015c0d1a134a33d2008775f4d

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 271cca3564ceb0197cf51a615b99ea20a3aefb5ccadbf09d89ab28febc6aefda
MD5 140ec7ee7681eb3558a7921a206784b4
BLAKE2b-256 a036de8a0114a82f77a272ba04e7f07f31ed5773a48e59f8849db73b309ff4a1

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 506d60e1a267d22915f8e1afc536c6af7ea3d29b2136faa93bfa9e7d6938a0ec
MD5 afab564f6e2ce32d16cecf25083f7293
BLAKE2b-256 728d32f122d76a6445a018d789d4b5c46e795600e8cf4a367c9fae473a027e93

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 a90e6b793e765b6cb0e524cb74c919d62efa3114b3c98b7438d737a5cd1ae4ec
MD5 a48443284663f8d7ce4f4810abcc4610
BLAKE2b-256 ce80cd520c761242705f415b4f59a0d09cd1a05694423eff9c6a8f5d0cce3457

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 1f46da238ef642a04cdcceb98e2f5b03b4386cab916df0192b89c961130e0bcd
MD5 896b18bc6c4701a92200fb6031a93ed0
BLAKE2b-256 4f1c16762eab0bda2316dfe55591e2fc5b0cd0001339aebee082c814d73cc909

See more details on using hashes here.

File details

Details for the file tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for tensorflow_io_gcs_filesystem_nightly-0.23.0.dev20211214202800-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 b6993633005024a2dcfbda66bdcdd0bee3611eefff6e174975829d49c4353089
MD5 a2ffbdf83968118527838c8c344941a1
BLAKE2b-256 408dc256dee02b730d618f9a53902c3b859724d8a76cba1b1d31844dd10b8ec3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page