TensorFlow Transform
TensorFlow Transform is a library for preprocessing data with TensorFlow.
tf.Transform is useful for data that requires a full-pass, such as:
- Normalize an input value by mean and standard deviation.
- Convert strings to integers by generating a vocabulary over all input values.
- Convert floats to integers by assigning them to buckets based on the observed data distribution.
TensorFlow has built-in support for manipulations on a single example or a batch
of examples. tf.Transform extends these capabilities to support full-passes
over the example data.
The output of tf.Transform is exported as a
TensorFlow graph to use for training and
serving. Using the same graph for both training and serving can prevent skew
since the same transformations are applied in both stages.
For an introduction to tf.Transform, see the tf.Transform section of the
TFX Dev Summit talk on TFX
(link).
Installation
The tensorflow-transform
PyPI package is the
recommended way to install tf.Transform:
pip install tensorflow-transform
Build TFT from source
To build from source follow the following steps: Create a virtual environment by running the commands
python -m venv <virtualenv_name>
source <virtualenv_name>/bin/activate
git clone https://github.com/tensorflow/transform.git
cd transform
pip install .
If you are doing development on the TFT repo, replace
pip install .
with
pip install -e .
The -e flag causes TFT to be installed in development mode.
Nightly Packages
TFT also hosts nightly packages at https://pypi-nightly.tensorflow.org on Google Cloud. To install the latest nightly package, please use the following command:
pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple tensorflow-transform
This will install the nightly packages for the major dependencies of TFT such as TensorFlow Metadata (TFMD), TFX Basic Shared Libraries (TFX-BSL).
Running Tests
To run TFT tests, run the following command from the root of the repository:
pytest
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient distributed computation is supported. By default, Apache Beam runs in local mode but can also run in distributed mode using Google Cloud Dataflow and other Apache Beam runners.
Apache Arrow is also required. TFT uses Arrow to represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table is the tf.Transform package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
| tensorflow-transform | apache-beam[gcp] | pyarrow | tensorflow | tensorflow-metadata | tfx-bsl |
|---|---|---|---|---|---|
| GitHub master | 2.72.0 | 14.0.0 | nightly (2.x) | 1.21.0 | 1.21.0 |
| 1.21.0 | 2.72.0 | 14.0.0 | 2.21 | 1.21.0 | 1.21.0 |
| 1.17.0 | 2.65.0 | 10.0.1 | 2.17 | 1.17.1 | 1.17.1 |
| 1.16.0 | 2.60.0 | 10.0.1 | 2.16 | 1.16.1 | 1.16.1 |
| 1.15.0 | 2.47.0 | 10.0.0 | 2.15 | 1.15.0 | 1.15.1 |
| 1.14.0 | 2.47.0 | 10.0.0 | 2.13 | 1.14.0 | 1.14.0 |
| 1.13.0 | 2.41.0 | 6.0.0 | 2.12 | 1.13.1 | 1.13.0 |
| 1.12.0 | 2.41.0 | 6.0.0 | 2.11 | 1.12.0 | 1.12.0 |
| 1.11.0 | 2.41.0 | 6.0.0 | 1.15.5 / 2.10 | 1.11.0 | 1.11.0 |
| 1.10.0 | 2.40.0 | 6.0.0 | 1.15.5 / 2.9 | 1.10.0 | 1.10.0 |
| 1.9.0 | 2.38.0 | 5.0.0 | 1.15.5 / 2.9 | 1.9.0 | 1.9.0 |
| 1.8.0 | 2.38.0 | 5.0.0 | 1.15.5 / 2.8 | 1.8.0 | 1.8.0 |
| 1.7.0 | 2.36.0 | 5.0.0 | 1.15.5 / 2.8 | 1.7.0 | 1.7.0 |
| 1.6.1 | 2.35.0 | 5.0.0 | 1.15.5 / 2.8 | 1.6.0 | 1.6.0 |
| 1.6.0 | 2.35.0 | 5.0.0 | 1.15.5 / 2.7 | 1.6.0 | 1.6.0 |
| 1.5.0 | 2.34.0 | 5.0.0 | 1.15.2 / 2.7 | 1.5.0 | 1.5.0 |
| 1.4.1 | 2.33.0 | 4.0.1 | 1.15.2 / 2.6 | 1.4.0 | 1.4.0 |
| 1.4.0 | 2.33.0 | 4.0.1 | 1.15.2 / 2.6 | 1.4.0 | 1.4.0 |
| 1.3.0 | 2.31.0 | 2.0.0 | 1.15.2 / 2.6 | 1.2.0 | 1.3.0 |
| 1.2.0 | 2.31.0 | 2.0.0 | 1.15.2 / 2.5 | 1.2.0 | 1.2.0 |
| 1.1.1 | 2.29.0 | 2.0.0 | 1.15.2 / 2.5 | 1.1.0 | 1.1.1 |
| 1.1.0 | 2.29.0 | 2.0.0 | 1.15.2 / 2.5 | 1.1.0 | 1.1.0 |
| 1.0.0 | 2.29.0 | 2.0.0 | 1.15 / 2.5 | 1.0.0 | 1.0.0 |
| 0.30.0 | 2.28.0 | 2.0.0 | 1.15 / 2.4 | 0.30.0 | 0.30.0 |
| 0.29.0 | 2.28.0 | 2.0.0 | 1.15 / 2.4 | 0.29.0 | 0.29.0 |
| 0.28.0 | 2.28.0 | 2.0.0 | 1.15 / 2.4 | 0.28.0 | 0.28.1 |
| 0.27.0 | 2.27.0 | 2.0.0 | 1.15 / 2.4 | 0.27.0 | 0.27.0 |
| 0.26.0 | 2.25.0 | 0.17.0 | 1.15 / 2.3 | 0.26.0 | 0.26.0 |
| 0.25.0 | 2.25.0 | 0.17.0 | 1.15 / 2.3 | 0.25.0 | 0.25.0 |
| 0.24.1 | 2.24.0 | 0.17.0 | 1.15 / 2.3 | 0.24.0 | 0.24.1 |
| 0.24.0 | 2.23.0 | 0.17.0 | 1.15 / 2.3 | 0.24.0 | 0.24.0 |
| 0.23.0 | 2.23.0 | 0.17.0 | 1.15 / 2.3 | 0.23.0 | 0.23.0 |
| 0.22.0 | 2.20.0 | 0.16.0 | 1.15 / 2.2 | 0.22.0 | 0.22.0 |
| 0.21.2 | 2.17.0 | 0.15.0 | 1.15 / 2.1 | 0.21.0 | 0.21.3 |
| 0.21.0 | 2.17.0 | 0.15.0 | 1.15 / 2.1 | 0.21.0 | 0.21.0 |
| 0.15.0 | 2.16.0 | 0.14.0 | 1.15 / 2.0 | 0.15.0 | 0.15.0 |
| 0.14.0 | 2.14.0 | 0.14.0 | 1.14 | 0.14.0 | n/a |
| 0.13.0 | 2.11.0 | n/a | 1.13 | 0.12.1 | n/a |
| 0.12.0 | 2.10.0 | n/a | 1.12 | 0.12.0 | n/a |
| 0.11.0 | 2.8.0 | n/a | 1.11 | 0.9.0 | n/a |
| 0.9.0 | 2.6.0 | n/a | 1.9 | 0.9.0 | n/a |
| 0.8.0 | 2.5.0 | n/a | 1.8 | n/a | n/a |
| 0.6.0 | 2.4.0 | n/a | 1.6 | n/a | n/a |
| 0.5.0 | 2.3.0 | n/a | 1.5 | n/a | n/a |
| 0.4.0 | 2.2.0 | n/a | 1.4 | n/a | n/a |
| 0.3.1 | 2.1.1 | n/a | 1.3 | n/a | n/a |
| 0.3.0 | 2.1.1 | n/a | 1.3 | n/a | n/a |
| 0.1.10 | 2.0.0 | n/a | 1.0 | n/a | n/a |
Questions
Please direct any questions about working with tf.Transform to
Stack Overflow using the
tensorflow-transform
tag.
2 maintainers
Release files for tensorflow-transform 1.21.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tensorflow_transform-1.21.0.tar.gz | 394.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tensorflow_transform-1.21.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 856.7 kB
Release files / tensorflow_transform-1.21.0.tar.gz
| Download URL | tensorflow_transform-1.21.0.tar.gz |
|---|---|
| Size | 394.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6f0ad1efc985ab8138c1f84f3f625ecc5e989a39e5a881afcbb3a2348fbc14b1
|
|
BLAKE2b-256 checksum How to use checksums |
fec8fbfecd0df52fcffacfe890894f679dcc6dfcdbd05690e3ba4965d20477fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / tensorflow_transform-1.21.0-py3-none-any.whl
| Download URL | tensorflow_transform-1.21.0-py3-none-any.whl |
|---|---|
| Size | 462.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
23346f223793f4b03612d0b9f85677a64104f148c8d39cc120696ccdfe6d3aea
|
|
BLAKE2b-256 checksum How to use checksums |
30765a200c421a641b89c06371928f4f379f96021d53604efb5b734921ba5fc1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
