towhee 0.8.0

Towhee is a framework that helps you encode your unstructured data into embeddings.

 

x2vec, Towhee is all you need!

ENGLISH | 中文文档

 

Towhee makes it easy to build neural data processing pipelines for AI applications. We provide several hundred models, algorithms, and transformations as standard pipeline building blocks. You can prototype your pipeline with our Pythonic API, and use Towhee to automatically optimize it for production-ready environments.

:art: Various Modalities: We support data processing on different modalities, such as images, videos, text, audio, molecular structures, etc.

:mortar_board: SOTA Models: We provide SOTA models across 5 fields (CV, NLP, Multimodal, Audio, Medical), 15 tasks, 140+ model architectures, 700+ pretrained models. These include BERT, CLIP, ViT, SwinTransformer, MAE, data2vec, etc.

:package: Data Processing: Towhee also provides traditional data processing methods that can be used together with neural network models to help you build practical data processing pipelines. Video decoding, audio slicing, frame sampling, feature vector dimension reduction, model ensemble, and database operations are a small sample of the different operators we provide.

:snake: Pythonic API: Towhee includes a pythonic method-chaining API for describing custom data processing pipelines. We also support schemas, making processing unstructured data as easy as handling tabular data.

What's New

v0.7.3 Jul.27,2022

• Add one multimodal (text/image) model: CoCa.
• Add two video models for grounded situation recognition & repetitive action counting: CoFormer, TransRAC.
• Add two SoTA models for image tasks (image retrieval, image classification, etc.): CVNet, MaxViT

v0.7.1 Jul.1,2022

• Add one image embedding model: MPViT.
• Add two video retrieval models: BridgeFormer, collaborative-experts.
• Add FAISS-based ANNSearch operators: to_faiss, faiss_search.

v0.7.0 Jun.24,2022

• Add six video understanding/classification models: Video Swin Transformer, TSM, Uniformer, OMNIVORE, TimeSformer, MoViNets.
• Add four video retrieval models: CLIP4Clip, DRL, Frozen in Time, MDMMT.

v0.6.1 May.13,2022

• Add three text-image retrieval models: CLIP, BLIP, LightningDOT.
• Add six video understanding/classification models from PyTorchVideo: I3D, C2D, Slow, SlowFast, X3D, MViT.

Getting started

Towhee requires Python 3.6+. Towhee can be installed via pip:

pip install towhee towhee.models

If you run into any pip-related install problems, please try to upgrade pip with pip install -U pip.

Try your first Towhee pipeline. In this example, we show how to create a CLIP-based cross modal retrieval pipeline within 15 lines of code.

import towhee

# create image embeddings and build index
(
    towhee.glob['file_name']('./*.png')
          .image_decode['file_name', 'img']()
          .image_text_embedding.clip['img', 'vec'](model_name='clip_vit_b32', modality='image')
          .tensor_normalize['vec','vec']()
          .to_faiss[('file_name', 'vec')](findex='./index.bin')
)

# search image by text
results = (
    towhee.dc['text'](['puppy Corgi'])
          .image_text_embedding.clip['text', 'vec'](model_name='clip_vit_b32', modality='text')
          .tensor_normalize['vec', 'vec']()
          .faiss_search['vec', 'results'](findex='./index.bin', k=3)
          .select['text', 'results']()
)

Learn more examples from Towhee bootcamp

Core Concepts

Towhee is composed of four main building blocks - Operators, Pipelines, DataCollection API and Engine.

• Operator: An operator is a single building block of neural data processing pipelines. Different implementations of operators are categorized by tasks, with standard task interface. An operator can be a deep learning model, a data processing method, or a Python function.

• Pipeline: A pipeline is composed of several operators. Operators are connected together as a DAG(directed acyclic graph) to create complex functionalities, such as embedding feature extraction, data tagging, cross modal data understanding, etc.

• DataCollection: A pythonic and method-chaining style API that for building custom pipelines. Pipelines defined by DataColltion API can be either run on notebook locally for fast prototyping, or converting to image docker with end-to-end optimization for production-ready environments.

• Engine: The engine sits at Towhee's core. Given a pipeline, the engine will drive dataflow between individual operators, schedule tasks, and monitor compute resource (CPU/GPU/etc) usage. We provide a basic engine within Towhee to run pipelines on a single-instance machine, and Triton-based engine to run pipelines in docker containers.

Contributing

Remember that writing code is not the only way to contribute! Submitting issues, answering questions, and improving documentation are some of the many ways you can join our growing community. Check out our contributing page for more information.

Special thanks goes to these folks for contributing to Towhee, either on Github, our Towhee Hub, or elsewhere:

Looking for a database to store and index your embedding vectors? Check out Milvus.