PDF and Web Content Query Package

This package provides functionality to process PDF files and web pages, allowing users to query their content using natural language processing techniques.

Table of Contents

• Features
• Installation
• Usage
  • Processing a PDF
  • Crawling and Querying a Web Page
• How It Works
• Dependencies
• License
• Contributing
• Support

Features

• Process PDF files and answer queries about their content
• Crawl web pages and answer queries about their content
• Utilizes advanced embedding techniques for accurate content matching

Installation

To install this package, run:

pip install semanticbot

Replace semanticbot with the actual name of your package.

Usage

Processing a PDF

To process a PDF file and query its content:

from your_package_name import process_pdf

pdf_path = "path/to/your/file.pdf"
query = "What is the main topic of this document?"

results = process_pdf(pdf_path, query)

for chunk, similarity in results:
    print(f"Similarity: {similarity}")
    print(f"Text chunk: {chunk}
")

Crawling and Querying a Web Page

To crawl a web page and query its content:

from your_package_name import crawl_and_query

url = "https://example.com"
query = "What are the key features of the product?"

results = crawl_and_query(url, query)

for chunk, similarity in results:
    print(f"Similarity: {similarity}")
    print(f"Text chunk: {chunk}
")

How It Works

• For PDFs: The package extracts text content from the file.
• For Web Pages: It crawls the specified URL and extracts the text content.
• The extracted text is split into manageable chunks.
• The package uses HuggingFace's BGE embeddings to convert text chunks and the query into vector representations.
• Cosine similarity is used to find the most relevant text chunks for the given query.
• The top 5 most relevant chunks are returned along with their similarity scores.

License

This project is licensed under the MIT License - see the LICENSE file for details.