Pre-release
This release is a pre-release and may not be stable for production use.
Khoj 🦅
A natural language search engine for your personal notes, transactions and images
Table of Contents
- Features
- Demo
- Architecture
- Setup
- Use
- Upgrade
- Troubleshoot
- Miscellaneous
- Development
- Performance
- Credits
Features
- Natural: Advanced Natural language understanding using Transformer based ML Models
- Local: Your personal data stays local. All search, indexing is done on your machine*
- Incremental: Incremental search for a fast, search-as-you-type experience
- Pluggable: Modular architecture makes it easy to plug in new data sources, frontends and ML models
- Multiple Sources: Search your Org-mode and Markdown notes, Beancount transactions and Photos
- Multiple Interfaces: Search using a Web Browser, Emacs or the API
Demo
https://user-images.githubusercontent.com/6413477/181664862-31565b0a-0e64-47e1-a79a-599dfc486c74.mp4
Description
- User searches for "Setup editor"
- The demo looks for the most relevant section in this readme and the khoj.el readme
- Top result is what we are looking for, the section to Install Khoj.el on Emacs
Analysis
- The results do not have any words used in the query
- Based on the top result it seems the re-ranking model understands that Emacs is an editor?
- The results incrementally update as the query is entered
- The results are re-ranked, for better accuracy, once user is idle
Architecture
Setup
1. Install
pip install khoj-assistant
2. Configure
- Set
input-filesorinput-filterin each relevantcontent-typesection of khoj_sample.yml- Set
input-directoriesfield incontent-type.imagesection
- Set
- Delete
content-typesections irrelevant for your use-case
3. Run
khoj config/khoj_sample.yml -vv
Loads ML model, generates embeddings and exposes API to search notes, images, transactions etc specified in config YAML
Use
- Khoj via Web
- Khoj via Emacs
- Khoj via API
Upgrade
pip install --upgrade khoj-assistant
Troubleshoot
-
Symptom: Errors out complaining about Tensors mismatch, null etc
- Mitigation: Delete
content-type>imagesection fromkhoj_sample.yml
- Mitigation: Delete
-
Symptom: Errors out with "Killed" in error message in Docker
- Fix: Increase RAM available to Docker Containers in Docker Settings
- Refer: StackOverflow Solution, Configure Resources on Docker for Mac
Miscellaneous
- The experimental chat API endpoint uses the OpenAI API
- It is disabled by default
- To use it add your
openai-api-keyto config.yml
Development
Setup
Using Docker
- Clone
git clone https://github.com/debanjum/khoj && cd khoj
- Configure
- Required: Update docker-compose.yml to mount your images, (org-mode or markdown) notes and beancount directories
- Optional: Edit application configuration in khoj_docker.yml
- Run
docker-compose up -d
Note: The first run will take time. Let it run, it's mostly not hung, just generating embeddings
Using Conda
-
Install Dependencies
- Install Conda [Required]
- Install Exiftool [Optional]
sudo apt -y install libimage-exiftool-perl
-
Install Khoj
git clone https://github.com/debanjum/khoj && cd khoj conda env create -f config/environment.yml conda activate khoj
-
Configure
- Set
input-filesorinput-filterin each relevantcontent-typesection ofkhoj_sample.yml- Set
input-directoriesfield inimagecontent-typesection
- Set
- Delete
content-typesections irrelevant for your use-case
- Set
-
Run Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
python3 -m src.main -c=config/khoj_sample.yml -vv
Upgrade
Using Docker
docker-compose build --pull
Using Conda
cd khoj
git pull origin master
conda deactivate khoj
conda env update -f config/environment.yml
conda activate khoj
Test
pytest
Performance
Query performance
- Semantic search using the bi-encoder is fairly fast at <5 ms
- Reranking using the cross-encoder is slower at <2s on 15 results. Tweak
top_kto tradeoff speed for accuracy of results - Applying explicit filters is very slow currently at ~6s. This is because the filters are rudimentary. Considerable speed-ups can be achieved using indexes etc
Indexing performance
- Indexing is more strongly impacted by the size of the source data
- Indexing 100K+ line corpus of notes takes 6 minutes
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
- Once https://github.com/debanjum/khoj/issues/36 is implemented, it should only take this long on first run
Miscellaneous
- Testing done on a Mac M1 and a >100K line corpus of notes
- Search, indexing on a GPU has not been tested yet
Credits
- Multi-QA MiniLM Model, All MiniLM Model for Text Search. See SBert Documentation
- OpenAI CLIP Model for Image Search. See SBert Documentation
- Charles Cave for OrgNode Parser
- Org.js to render Org-mode results on the Web interface
- Markdown-it to render Markdown results on the Web interface
- Sven Marnach for PyExifTool
Release files for khoj-assistant 0.1.5a1659657577
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| khoj-assistant-0.1.5a1659657577.tar.gz | 163.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| khoj_assistant-0.1.5a1659657577-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 336.3 kB
Release files / khoj-assistant-0.1.5a1659657577.tar.gz
| Download URL | khoj-assistant-0.1.5a1659657577.tar.gz |
|---|---|
| Size | 163.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
86421b6f4d1d5776bb08d6f9ff40971d49662705bbeb66502e8f15a9935ab24f
|
|
BLAKE2b-256 checksum How to use checksums |
f888afec8c7dd14c651890ffc2ddc612992890ab7180a5b0392d224905e83f72
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.10.5
|
Release files / khoj_assistant-0.1.5a1659657577-py3-none-any.whl
| Download URL | khoj_assistant-0.1.5a1659657577-py3-none-any.whl |
|---|---|
| Size | 172.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8394445f29b80dcec4e1055bdb2bd008af4afcd8d764ef246bf82fcca1d11d96
|
|
BLAKE2b-256 checksum How to use checksums |
13192e3b059bc45f2a71ab3d6aa011b97f4a0b047df7b0cbac4074435092cb44
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.10.5
|
