chonky-py 0.1.0

Chonky-Py

Features · Modes · Setup · Installation · Loading · Processing · Analysis · Tokens · Embeddings · Vectors · Search · Requirements ·

Chonky is a modular Python and Streamlit text-processing application for analysts, data scientists, and machine-learning practitioners working with unstructured, semi-structured, and document-based text. It provides a complete pipeline for document ingestion, text cleaning, NLP preprocessing, token diagnostics, semantic analysis, embedding generation, vector persistence, and similarity retrieval.

The application is designed around an explicit, staged workflow. Users load documents from local, web, email, cloud, notebook, public-data, and repository sources; clean and normalize text; tokenize and inspect corpus structure; generate embeddings through hosted providers; persist vectors with

sqlite-vec; and run semantic similarity search over stored chunks.

PyPI installation

python -m pip install chonky-py
chonky

For the Windows tabless desktop window (WebView2 Runtime required):

python -m pip install "chonky-py[desktop]"
chonky-desktop

Python 3.11 or later is required. The distribution is chonky-py; the import namespace is chonky. Do not install it alongside the unrelated PyPI chonky package in the same environment. Provider credentials and external NLP/model resources remain separate. Publication steps are in the PyPI release guide.

🎥 Demo

☁️ Cloud

🧠 Features

🧭 Application Tabs

The current Streamlit app is organized into the following tabs.

🏛 Architecture

Document Sources
    │
    ├── Local Files / Corpora / Email / Notebooks
    ├── Web / Wikipedia / ArXiv / GitHub / PubMed / Open City
    └── Cloud Files / Buckets / OneDrive / SharePoint
            │
            ▼
      Loading Tab
            │ raw_text + documents
            ▼
      Text Processing Tab
            │ processed_text
            ▼
      Semantic Analysis Tab
            │ chunks + vocabulary + frequencies
            ▼
      Data Tokenization Tab
            │ sentence rows + token grids + readiness diagnostics
            ▼
      Tensor Embeddings Tab
            │ embeddings + provider/model metadata
            ▼
      Vector Database Tab
            │ sqlite-vec tables + similarity search
            ▼
      Retrieval / RAG-Ready Outputs

🧰 Setup Instructions

⚡ Clone the Repository

git clone https://github.com/is-leeroy-jenkins/Chonky.git
cd Chonky
python -m venv .venv

▶️ Activate the Environment

Windows PowerShell

.\.venv\Scripts\Activate.ps1

Windows Command Prompt

.\.venv\Scripts\activate.bat

macOS / Linux

source .venv/bin/activate

📦 Install Requirements

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

🚀 Run Chonky

python -m streamlit run app.py

🔑 API Key Setup

The sidebar exposes API key inputs for hosted embedding, cloud, and future vector-service workflows. Set only the providers you plan to use.

Windows PowerShell Example

$env:OPENAI_API_KEY="your_openai_key"
$env:GEMINI_API_KEY="your_gemini_key"
$env:GROQ_API_KEY="your_groq_key"
$env:GOOGLE_API_KEY="your_google_key"

macOS / Linux Example

export OPENAI_API_KEY="your_openai_key"
export GEMINI_API_KEY="your_gemini_key"
export GROQ_API_KEY="your_groq_key"
export GOOGLE_API_KEY="your_google_key"

🧪 Example Usage

Basic Text Processing

from processors import TextParser

processor = TextParser()
raw_text = "<p>This is an example document.</p>"
clean_text = processor.remove_html(raw_text)
clean_text = processor.normalize_text(clean_text)
clean_text = processor.collapse_whitespace(clean_text)
print(clean_text)

Chunk Text for Embedding

from nltk.tokenize import word_tokenize

text = "This is a processed document ready for semantic analysis."
tokens = word_tokenize(text)
chunks = [" ".join(tokens[index:index + 100]) for index in range(0, len(tokens), 100)]
print(chunks)

Similarity Retrieval with sqlite-vec

import sqlite3
import sqlite_vec
from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from langchain_community.vectorstores import SQLiteVec

conn = sqlite3.connect("vectors.db")
conn.enable_load_extension(True)
sqlite_vec.load(conn)

embedding_fn = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")

vector_store = SQLiteVec(
    connection=conn,
    table_name="example__OpenAI__text-embedding-3-small__1536",
    embedding=embedding_fn,
)

results = vector_store.similarity_search(query="What is this document about?", k=5)

for document in results:
    print(document.page_content)

📥 Document Loading

The Loading tab is organized into local, web, and cloud document groups. Each loader normalizes content into LangChain Document objects and updates the shared raw-text buffer for downstream processing.

Local Documents

Web Documents

Cloud Documents

🧼 Text Processing

Text Processing consumes raw_text and produces the authoritative processed_text used by downstream semantic analysis, tokenization, embeddings, and vector persistence.

🔎 Semantic Analysis

Semantic Analysis requires processed text. It chunks text, computes lexical structures, and prepares intermediate outputs for diagnostics and embedding workflows.

🔤 Data Tokenization

The Data Tokenization tab turns processed text into row-based and grid-based diagnostic structures. It is intended to reveal whether text is ready for vectorization.

🧠 Tensor Embeddings

The Tensor Embeddings tab generates embeddings from either the full processed text or generated chunked documents.

Embedding Diagnostics

🗄️ Vector Database

The Vector Database tab persists generated embeddings to SQLite using sqlite-vec and LangChain's SQLiteVec vector store integration.

Vector tables are named deterministically using:

<document>__<provider>__<model>__<dimension>

This prevents accidental mixing of embeddings generated by different providers, models, or vector dimensions.

🔍 Similarity Search

Chonky includes an interactive sqlite-vec similarity search interface.

📦 Requirements

The table below reflects the active imports and runtime features used by the current app.py. Use the repository requirements.txt as the installation source of truth when version pins are present.

📝 License

Chonky is published under the MIT License.

🙏 Acknowledgements

• Streamlit
• NLTK
• LangChain
• sqlite-vec
• OpenAI
• Google Gemini
• Hugging Face sentence-transformers
• The open-source Python NLP and machine-learning ecosystem