A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster
Project description
HybridBackend
HybridBackend is a high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster.
Features
-
Memory-efficient loading of categorical data
-
Communication-efficient training and evaluation at scale
-
GPU-efficient orchestration of embedding layers
-
Easy to use with existing AI workflows
Install
Using container images
Linux Distro | CUDA | Python | Tensorflow | URL |
---|---|---|---|---|
Ubuntu 20.04 | 11.4 | 3.8 | 1.15.5 | registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:0.6-tf1.15-py3.8-cu114-ubuntu20.04 |
See PAI DLC for more information.
Using pip packages
CUDA 11.4
Features:
All
Requires:
- NVIDIA TensorFlow 1.15
- Python 3.8
- CUDA 11.4 or above
- Ubuntu 20.04 or above
pip install nvidia-pyindex
pip install hybridbackend-tf115-cu114
CUDA 10.0
Features:
- Memory-efficient loading of categorical data
- GPU-efficient orchestration of embedding layers
Requires:
- TensorFlow GPU 1.15
- Python 3.6
- CUDA 10.0 or above
- Ubuntu 18.04 or above
pip install hybridbackend-tf115-cu100
CPU
Features:
- Memory-efficient loading of categorical data
Requires:
- TensorFlow 1.15
- Python 3.6
- Ubuntu 18.04 or above
pip install hybridbackend-tf115-cpu
Build from source
Usage
A minimal example:
import tensorflow as tf
import hybridbackend.tensorflow as hb
def model_fn(features, labels, mode, params):
# ..
dense_features = hb.keras.layers.DenseFeatures(columns)
# ...
# ...
estimator = hb.estimator.Estimator(model_fn, model_dir=model_dir)
estimator.train_and_evaluate(train_spec, eval_spec)
Please see documentation for more information.
License
HybridBackend is licensed under the Apache 2.0 License.
Community
-
Please see Contributing Guide before your first contribution.
-
Please register as an adopter if your organization is interested in adoption. We will discuss RoadMap with registered adopters in advance.
-
Please cite HybridBackend in your publications if it helps:
@inproceedings{zhang2022picasso, title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems}, author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others}, booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)}, year={2022}, organization={IEEE} }
Contact Us
If you would like to share your experiences with others, you are welcome to contact us in DingTalk:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for hybridbackend_tf115_cpu-0.6.1a0.dev20220727110303-cp36-cp36m-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a8330d22223a6103ee94bfefd0da67d368d2c692c4eb6afcbebbf1431e69265 |
|
MD5 | 93280b6c1e3bee04a4f8f40ef2282740 |
|
BLAKE2b-256 | 05fd03f75c984c679e47f212e9570c6b73bea619328dc8314256a6b561e182cd |