semantic-search-nollms 0.2

No project description provided

AI-Powered Document Semantic Search with Web Crawling

This project implements an AI-powered document semantic search system using Streamlit, LangChain, and various document processing techniques. It allows users to upload documents, scan images, crawl web pages, process the content, and perform semantic searches without relying on a large language model.

Table of Contents

• Features
• System Architecture
• Setup
• Usage
• Features
• Contributing
• License

Features

• Document upload and processing (PDF, DOCX, TXT, XLS, XLSX)
• Image scanning and OCR for text extraction
• Web crawling for content retrieval
• Text chunking and embedding using Hugging Face's sentence transformers
• FAISS vector store for efficient similarity search
• Cosine similarity-based semantic search without LLM
• Customizable chunk size and overlap
• Streamlit-based user interface for easy interaction

Setup

1. Clone the repository:

   https://github.com/FastianAbdullah/Semantic-Search-Without-LLM.git
   

2. Install the required dependencies:

   pip install -r requirements.txt
   

Usage:

To start the application, run:

streamlit run doc_search_and_crawl.py

The application will open in your default web browser.

API Endpoints

This project is primarily a Streamlit web application and does not expose traditional API endpoints. However, the main functionalities are accessible through the Streamlit UI:

1. Document Upload

• Upload documents (PDF, DOCX, TXT, XLS, XLSX) using the file uploader in the "Document Upload" tab.

2. Document Scanning

• Use your camera to scan documents in the "Document Scan" tab.

3. Semantic Search

• Enter your query in the search bar after processing a document or scanned image.

4. Web Crawling

• Enter a URL in the "Web Crawl" tab to fetch and process web content.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

View Acknowledgements

• Streamlit for the web app framework
• LangChain for document processing utilities
• Hugging Face for sentence transformers
• FAISS for efficient similarity search
• BeautifulSoup for web scraping

Dependencies

View Dependencies

• streamlit
• langchain-text-splitters
• langchain-community
• sentence-transformers
• faiss-cpu
• pdfplumber
• pandas
• docx2txt
• numpy
• pillow
• pytesseract
• PyPDF2

Code Overview

The main script doc_search_and_crawl.py contains the following key components:

View Components

1. DataLoader class: Handles document loading and chunking for various file types.
2. WebCrawler class: Handles web content retrieval and parsing.
3. cosine_similarity function: Calculates the cosine similarity between two vectors.
4. scan_document function: Uses OCR to extract text from scanned images.
5. main function: Sets up the Streamlit interface and manages the overall flow of the application.
6. process_chunks function: Creates embeddings, builds the FAISS index, and performs the semantic search.

Future Improvements

View Future Improvements

• Add support for more file types
• Implement multi-language support
• Optimize performance for larger documents
• Integrate with cloud storage services for document management
• Implement advanced web crawling features (depth control, multiple page crawling)

Feel free to contribute to these improvements or suggest new features!