TensorFlow Data Validation

Python PyPI Documentation

TensorFlow Data Validation (TFDV) is a library for exploring and validating machine learning data. It is designed to be highly scalable and to work well with TensorFlow and TensorFlow Extended (TFX).

TF Data Validation includes:

  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well as faceted comparison of pairs of features (Facets)
  • Automated data-schema generation to describe expectations about data like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies, such as missing features, out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and learn more in order to correct them.

For instructions on using TFDV, see the get started guide and try out the example notebook. Some of the techniques implemented in TFDV are described in a technical paper published in SysML'19.

Installing from PyPI

The recommended way to install TFDV is using the PyPI package:

pip install tensorflow-data-validation

Nightly Packages

TFDV also hosts nightly packages on Google Cloud. To install the latest nightly package, please use the following command:

export TFX_DEPENDENCY_SELECTOR=NIGHTLY
pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation

This will install the nightly packages for the major dependencies of TFDV such as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).

Sometimes TFDV uses those dependencies' most recent changes, which are not yet released. Because of this, it is safer to use nightly versions of those dependent libraries when using nightly TFDV. Export the TFX_DEPENDENCY_SELECTOR environment variable to do so.

NOTE: These nightly packages are unstable and breakages are likely to happen. The fix could often take a week or more depending on the complexity involved.

Build with Docker

This is the recommended way to build TFDV under Linux, and is continuously tested at Google.

1. Install Docker

Please first install docker and docker-compose by following the directions: docker; docker-compose.

2. Clone the TFDV repository

git clone https://github.com/tensorflow/data-validation
cd data-validation

Note that these instructions will install the latest master branch of TensorFlow Data Validation. If you want to install a specific branch (such as a release branch), pass -b <branchname> to the git clone command.

3. Build the pip package

Then, run the following at the project root:

sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010

where PYTHON_VERSION is one of {39, 310, 311}.

A wheel will be produced under dist/.

4. Install the pip package

pip install dist/*.whl

Build from source

1. Prerequisites

To compile and use TFDV, you need to set up some prerequisites.

Install NumPy

If NumPy is not installed on your system, install it now by following these directions.

Install Bazel

If Bazel is not installed on your system, install it now by following these directions.

2. Clone the TFDV repository

git clone https://github.com/tensorflow/data-validation
cd data-validation

Note that these instructions will install the latest master branch of TensorFlow Data Validation. If you want to install a specific branch (such as a release branch), pass -b <branchname> to the git clone command.

3. Build the pip package

TFDV wheel is Python version dependent -- to build the pip package that works for a specific Python version, use that Python binary to run:

python setup.py bdist_wheel

You can find the generated .whl file in the dist subdirectory.

4. Install the pip package

pip install dist/*.whl

Supported platforms

TFDV is tested on the following 64-bit operating systems:

  • macOS 12.5 (Monterey) or later.
  • Ubuntu 20.04 or later.

Notable Dependencies

TensorFlow is required.

Apache Beam is required; it's the way that efficient distributed computation is supported. By default, Apache Beam runs in local mode but can also run in distributed mode using Google Cloud Dataflow and other Apache Beam runners.

Apache Arrow is also required. TFDV uses Arrow to represent data internally in order to make use of vectorized numpy functions.

Compatible versions

The following table shows the package versions that are compatible with each other. This is determined by our testing framework, but other untested combinations may also work.

tensorflow-data-validation apache-beam[gcp] pyarrow tensorflow tensorflow-metadata tensorflow-transform tfx-bsl
GitHub master 2.65.0 14.0.0 nightly (2.x) 1.21.0 n/a 1.21.0
1.21.0 2.65.0 14.0.0 2.21 1.21.0 n/a 1.21.0
1.17.0 2.65.0 10.0.1 2.17 1.17.1 n/a 1.17.1
1.16.1 2.59.0 10.0.1 2.16 1.16.1 n/a 1.16.1
1.16.0 2.59.0 10.0.1 2.16 1.16.0 n/a 1.16.0
1.15.1 2.47.0 10.0.0 2.15 1.15.0 n/a 1.15.1
1.15.0 2.47.0 10.0.0 2.15 1.15.0 n/a 1.15.0
1.14.0 2.47.0 10.0.0 2.13 1.14.0 n/a 1.14.0
1.13.0 2.40.0 6.0.0 2.12 1.13.1 n/a 1.13.0
1.12.0 2.40.0 6.0.0 2.11 1.12.0 n/a 1.12.0
1.11.0 2.40.0 6.0.0 1.15 / 2.10 1.11.0 n/a 1.11.0
1.10.0 2.40.0 6.0.0 1.15 / 2.9 1.10.0 n/a 1.10.1
1.9.0 2.38.0 5.0.0 1.15 / 2.9 1.9.0 n/a 1.9.0
1.8.0 2.38.0 5.0.0 1.15 / 2.8 1.8.0 n/a 1.8.0
1.7.0 2.36.0 5.0.0 1.15 / 2.8 1.7.0 n/a 1.7.0
1.6.0 2.35.0 5.0.0 1.15 / 2.7 1.6.0 n/a 1.6.0
1.5.0 2.34.0 5.0.0 1.15 / 2.7 1.5.0 n/a 1.5.0
1.4.0 2.32.0 4.0.1 1.15 / 2.6 1.4.0 n/a 1.4.0
1.3.0 2.32.0 2.0.0 1.15 / 2.6 1.2.0 n/a 1.3.0
1.2.0 2.31.0 2.0.0 1.15 / 2.5 1.2.0 n/a 1.2.0
1.1.1 2.29.0 2.0.0 1.15 / 2.5 1.1.0 n/a 1.1.1
1.1.0 2.29.0 2.0.0 1.15 / 2.5 1.1.0 n/a 1.1.0
1.0.0 2.29.0 2.0.0 1.15 / 2.5 1.0.0 n/a 1.0.0
0.30.0 2.28.0 2.0.0 1.15 / 2.4 0.30.0 n/a 0.30.0
0.29.0 2.28.0 2.0.0 1.15 / 2.4 0.29.0 n/a 0.29.0
0.28.0 2.28.0 2.0.0 1.15 / 2.4 0.28.0 n/a 0.28.1
0.27.0 2.27.0 2.0.0 1.15 / 2.4 0.27.0 n/a 0.27.0
0.26.1 2.28.0 0.17.0 1.15 / 2.3 0.26.0 0.26.0 0.26.0
0.26.0 2.25.0 0.17.0 1.15 / 2.3 0.26.0 0.26.0 0.26.0
0.25.0 2.25.0 0.17.0 1.15 / 2.3 0.25.0 0.25.0 0.25.0
0.24.1 2.24.0 0.17.0 1.15 / 2.3 0.24.0 0.24.1 0.24.1
0.24.0 2.23.0 0.17.0 1.15 / 2.3 0.24.0 0.24.0 0.24.0
0.23.1 2.24.0 0.17.0 1.15 / 2.3 0.23.0 0.23.0 0.23.0
0.23.0 2.23.0 0.17.0 1.15 / 2.3 0.23.0 0.23.0 0.23.0
0.22.2 2.20.0 0.16.0 1.15 / 2.2 0.22.0 0.22.0 0.22.1
0.22.1 2.20.0 0.16.0 1.15 / 2.2 0.22.0 0.22.0 0.22.1
0.22.0 2.20.0 0.16.0 1.15 / 2.2 0.22.0 0.22.0 0.22.0
0.21.5 2.17.0 0.15.0 1.15 / 2.1 0.21.0 0.21.1 0.21.3
0.21.4 2.17.0 0.15.0 1.15 / 2.1 0.21.0 0.21.1 0.21.3
0.21.2 2.17.0 0.15.0 1.15 / 2.1 0.21.0 0.21.0 0.21.0
0.21.1 2.17.0 0.15.0 1.15 / 2.1 0.21.0 0.21.0 0.21.0
0.21.0 2.17.0 0.15.0 1.15 / 2.1 0.21.0 0.21.0 0.21.0
0.15.0 2.16.0 0.14.0 1.15 / 2.0 0.15.0 0.15.0 0.15.0
0.14.1 2.14.0 0.14.0 1.14 0.14.0 0.14.0 n/a
0.14.0 2.14.0 0.14.0 1.14 0.14.0 0.14.0 n/a
0.13.1 2.11.0 n/a 1.13 0.12.1 0.13.0 n/a
0.13.0 2.11.0 n/a 1.13 0.12.1 0.13.0 n/a
0.12.0 2.10.0 n/a 1.12 0.12.1 0.12.0 n/a
0.11.0 2.8.0 n/a 1.11 0.9.0 0.11.0 n/a
0.9.0 2.6.0 n/a 1.9 n/a n/a n/a

Questions

Please direct any questions about working with TF Data Validation to Stack Overflow using the tensorflow-data-validation tag.

Links

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for google_opensource from gravatar.com google_opensource Avatar for tfx_oss from gravatar.com tfx_oss

Credits

Author: Google LLC

License

Apache Software License (Apache 2.0)

Requires

Python <4, >=3.10

Provides Extra

all dev docs mutual-information test visualization

Tags

tensorflow data validation tfx

Classifiers

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for google_opensource from gravatar.com google_opensource Avatar for tfx_oss from gravatar.com tfx_oss

Credits

Author: Google LLC

License

Apache Software License (Apache 2.0)

Requires

Python <4, >=3.10

Provides Extra

all dev docs mutual-information test visualization

Tags

tensorflow data validation tfx

Classifiers

Report project as malware

Release files for tensorflow-data-validation 1.21.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for tensorflow-data-validation 1.21.0
File
tensorflow_data_validation-1.21.0-cp313-cp313-manylinux_2_39_x86_64.whl 1.7 MB CPython 3.13 CPython 3.13 Linux glibc 2.39+ x86-64 Details
tensorflow_data_validation-1.21.0-cp313-cp313-macosx_11_0_arm64.whl 1.3 MB CPython 3.13 CPython 3.13 macOS 11.0+ ARM64 Details
tensorflow_data_validation-1.21.0-cp312-cp312-manylinux_2_39_x86_64.whl 1.7 MB CPython 3.12 CPython 3.12 Linux glibc 2.39+ x86-64 Details
tensorflow_data_validation-1.21.0-cp312-cp312-macosx_11_0_arm64.whl 1.3 MB CPython 3.12 CPython 3.12 macOS 11.0+ ARM64 Details
tensorflow_data_validation-1.21.0-cp311-cp311-manylinux_2_39_x86_64.whl 1.7 MB CPython 3.11 CPython 3.11 Linux glibc 2.39+ x86-64 Details
tensorflow_data_validation-1.21.0-cp311-cp311-macosx_11_0_arm64.whl 1.4 MB CPython 3.11 CPython 3.11 macOS 11.0+ ARM64 Details
tensorflow_data_validation-1.21.0-cp310-cp310-manylinux_2_39_x86_64.whl 1.7 MB CPython 3.10 CPython 3.10 Linux glibc 2.39+ x86-64 Details
tensorflow_data_validation-1.21.0-cp310-cp310-macosx_11_0_arm64.whl 1.3 MB CPython 3.10 CPython 3.10 macOS 11.0+ ARM64 Details

Total release size: 12.1 MB

Release files / tensorflow_data_validation-1.21.0-cp313-cp313-manylinux_2_39_x86_64.whl

Download URL tensorflow_data_validation-1.21.0-cp313-cp313-manylinux_2_39_x86_64.whl
Size 1.7 MB
Tags CPython 3.13 Linux glibc 2.39+ x86-64
SHA-256 checksum
How to use checksums 		182f6fe03684139b84cb25119ef27e4c034cb938c8b9795bc8821de6f9120b96
BLAKE2b-256 checksum
How to use checksums 		11afaf6fcbe5e3ec0b638257ffdd62bf62076cf8f40800c1b07fba23dd25ffc2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp313-cp313-macosx_11_0_arm64.whl

Download URL tensorflow_data_validation-1.21.0-cp313-cp313-macosx_11_0_arm64.whl
Size 1.3 MB
Tags CPython 3.13 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums 		f282e42668671803cbb3335d8271fd40fe7ec95e5ff276e6f4bac2d9793411d8
BLAKE2b-256 checksum
How to use checksums 		1ba7daf3dc1c2d242940250e7cee881a240397dcd09f8235a11186c71b76bb98
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp312-cp312-manylinux_2_39_x86_64.whl

Download URL tensorflow_data_validation-1.21.0-cp312-cp312-manylinux_2_39_x86_64.whl
Size 1.7 MB
Tags CPython 3.12 Linux glibc 2.39+ x86-64
SHA-256 checksum
How to use checksums 		e9bfef0ee8540458e35d9f6e69db2674afc20629ee872d2800ae93eddb8a3127
BLAKE2b-256 checksum
How to use checksums 		589acfc0e30d48e08e90660ebaec96e2c1a65506463427ef83f610c3a07e0375
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp312-cp312-macosx_11_0_arm64.whl

Download URL tensorflow_data_validation-1.21.0-cp312-cp312-macosx_11_0_arm64.whl
Size 1.3 MB
Tags CPython 3.12 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums 		b3f555acc54c6163eebdf69b0ab5965d1761b7e5c7a09b2156057e7f810eb296
BLAKE2b-256 checksum
How to use checksums 		405d085e15862e86c0cd3c070809e8e672662f66c383b42c5d64dbd8fa55c448
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp311-cp311-manylinux_2_39_x86_64.whl

Download URL tensorflow_data_validation-1.21.0-cp311-cp311-manylinux_2_39_x86_64.whl
Size 1.7 MB
Tags CPython 3.11 Linux glibc 2.39+ x86-64
SHA-256 checksum
How to use checksums 		6c148be69ae20c02f58b696efb343c4bb3e923184963f8b793afd0066c8ced9f
BLAKE2b-256 checksum
How to use checksums 		20819c568e9703698ba52ca889da5b6d61ee226c6d6b0ae2372cbead125e1b4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp311-cp311-macosx_11_0_arm64.whl

Download URL tensorflow_data_validation-1.21.0-cp311-cp311-macosx_11_0_arm64.whl
Size 1.4 MB
Tags CPython 3.11 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums 		230b390d2ee1aeab808123cc7ddf198fa2ffdb4e7b51fb75dd2f1e692ca456e0
BLAKE2b-256 checksum
How to use checksums 		24e9f735c55516d3a65e25011608b9d9cd75450f0d0c5903c09896cf6c1fcf2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp310-cp310-manylinux_2_39_x86_64.whl

Download URL tensorflow_data_validation-1.21.0-cp310-cp310-manylinux_2_39_x86_64.whl
Size 1.7 MB
Tags CPython 3.10 Linux glibc 2.39+ x86-64
SHA-256 checksum
How to use checksums 		0440f96763dc9612dc08a3c3cc66a23c5bedb0df781b7ea9d6da0b95afca05b2
BLAKE2b-256 checksum
How to use checksums 		011339bfed9c613efd26466d57393833effb4b4de4a47ac1886f3e882d4435af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release files / tensorflow_data_validation-1.21.0-cp310-cp310-macosx_11_0_arm64.whl

Download URL tensorflow_data_validation-1.21.0-cp310-cp310-macosx_11_0_arm64.whl
Size 1.3 MB
Tags CPython 3.10 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums 		7350c4f67819ce2445c33fc96101435847b46eec29de742a654d184c0460b1f9
BLAKE2b-256 checksum
How to use checksums 		47101973b24c554e1f35b24f905253717af5e91b34605822eee1996a0782c02e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/5.1.0 CPython/3.12.13

Release history Release notifications | RSS feed

This release

1.21.0 This release

8 release files

1.17.0

3 release files

1.16.1

6 release files

1.16.0

6 release files

1.15.1

6 release files

1.15.0

4 release files

1.14.0

9 release files

1.13.0

6 release files

1.12.0

9 release files

1.11.0

9 release files

1.10.0

9 release files

1.9.0

9 release files

1.8.0

9 release files

1.7.0

6 release files

1.6.0

6 release files

1.5.0

6 release files

1.4.0

6 release files

1.3.0

9 release files

1.2.0

9 release files

1.1.1

9 release files

1.1.0

9 release files

1.0.0

9 release files

0.30.0

9 release files

0.29.0

9 release files

0.28.0

9 release files

0.27.0

9 release files

0.26.1

9 release files

0.26.0

9 release files

0.25.0

9 release files

0.24.1

9 release files

0.24.0

9 release files

0.23.1

6 release files

0.23.0

9 release files

0.22.2

9 release files

0.22.1

9 release files

0.22.0

9 release files

0.21.5

11 release files

0.21.4

11 release files

0.21.2

11 release files

0.21.1

11 release files

0.21.0

11 release files

0.15.0

10 release files

0.14.1

10 release files

0.14.0

10 release files

0.13.1

8 release files

0.13.0

8 release files

0.12.0

3 release files

0.11.0

3 release files

0.9.0

3 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page