AI-powered search engine
txtai executes machine-learning workflows to transform data and build AI-powered text indices to perform similarity search.
Summary of txtai features:
- 🔎 Large-scale similarity search with multiple index backends (Faiss, Annoy, Hnswlib)
- 📄 Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors.
- 💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction
- ↪️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.
- 🔗 API bindings for JavaScript, Java, Rust and Go
- ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below:
| Application | Description |
|---|---|
| paperai | AI-powered literature discovery and review engine for medical/scientific papers |
| tldrstory | AI-powered understanding of headlines and story text |
| neuspo | Fact-driven, real-time sports event and news site |
| codequestion | Ask coding questions directly from the terminal |
txtai is built with Python 3.6+, Hugging Face Transformers, Sentence Transformers and FastAPI
Installation
The easiest way to install is via pip and PyPI
pip install txtai
Python 3.6+ is supported. Using a Python virtual environment is recommended.
See the detailed install instructions for more information covering installing from source, environment specific prerequisites and optional dependencies.
Examples
The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
Similarity Search
Build similarity/semantic/vector search applications.
| Notebook | Description | |
|---|---|---|
| Introducing txtai | Overview of the functionality provided by txtai |  |
| Build an Embeddings index with Hugging Face Datasets | Index and search Hugging Face Datasets | |
| Build an Embeddings index from a data source | Index and search a data source with word embeddings | |
| Add semantic search to Elasticsearch | Add semantic search to existing search systems | |
| API Gallery | Using txtai in JavaScript, Java, Rust and Go | |
| Similarity search with images | Embed images and text into the same space for search | |
| Distributed embeddings cluster | Distribute an embeddings index across multiple data nodes | |
Pipelines and Workflows
NLP-backed data transformation pipelines and workflows.
| Notebook | Description | |
|---|---|---|
| Extractive QA with txtai | Introduction to extractive question-answering with txtai | |
| Extractive QA with Elasticsearch | Run extractive question-answering queries with Elasticsearch | |
| Apply labels with zero shot classification | Use zero shot learning for labeling, classification and topic modeling | |
| Building abstractive text summaries | Run abstractive text summarization | |
| Extract text from documents | Extract text from PDF, Office, HTML and more | |
| Transcribe audio to text | Convert audio files to text | |
| Translate text between languages | Streamline machine translation and language detection | |
| Run pipeline workflows | Simple yet powerful constructs to efficiently process data | |
Model Training
Train NLP models.
| Notebook | Description | |
|---|---|---|
| Train a text labeler | Build text sequence classification models | |
| Train without labels | Use zero-shot classifiers to train new models | |
Applications
Series of example applications with txtai.
| Application | Description |
|---|---|
| Demo query shell | Basic similarity search example. Used in the original txtai demo. |
| Book search | Book similarity search application. Index book descriptions and query using natural language statements. |
| Image search | Image similarity search application. Index a directory of images and run searches to identify images similar to the input query. |
| Wiki search | Wikipedia search application. Queries Wikipedia API and summarizes the top result. |
| Workflow builder | Build and execute txtai workflows. Connect summarization, text extraction, transcription, translation and similarity search pipelines together to run unified workflows. |
Documentation
Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API.
Contributing
For those who would like to contribute to txtai, please see this guide.
1 maintainer
Release files for txtai 3.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| txtai-3.2.0.tar.gz | 43.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| txtai-3.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 111.3 kB
Release files / txtai-3.2.0.tar.gz
| Download URL | txtai-3.2.0.tar.gz |
|---|---|
| Size | 43.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d1857624970e3b0fd5a497afb2dac579ef47933ee25a252944af9e5cd7c2350e
|
|
BLAKE2b-256 checksum How to use checksums |
449a66eda76c21c611b9cb0074d66adee84efd2d18b037ff183929182ac31190
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.11
|
Release files / txtai-3.2.0-py3-none-any.whl
| Download URL | txtai-3.2.0-py3-none-any.whl |
|---|---|
| Size | 68.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
96af4098997828872f48b10b837aa8cda9bfdff9f25d060d08dd0abfe368e076
|
|
BLAKE2b-256 checksum How to use checksums |
bc8eee8d24dd0e549594f215d5be06611abd0bded00ceb35b122965f5892b3c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.11
|
