SynapseAI 0.5.1

No project description provided

PDF and Web Content Query Package With FAISS And Cosine Similarity

This package provides functionality to process PDF files and web pages, allowing users to query their content using natural language processing techniques.

Table of Contents

• Features
• Installation
• Usage
  • Processing a PDF
  • Crawling and Querying a Web Page
• How It Works
• Dependencies
• License
• Contributing
• Support

Features

• Process PDF files and answer queries about their content
• Crawl web pages and answer queries about their content
• Utilizes advanced embedding techniques for accurate content matching

Installation

To install this package, run:

pip install synapseai

Replace synapseai with the actual name of your package.

Usage

Processing a PDF

from synapseai import process_pdf

pdf_path = "path/to/your/document.pdf"
query = "What is the main topic of this document?"

results = process_pdf(pdf_path, query)

for chunk, similarity in results:
    print(f"Similarity: {similarity:.4f}")
    print(f"Chunk: {chunk[:200]}...")  # Print first 200 characters
    print("-" * 50)

Crawling and Querying a Web Page

To crawl a web page and query its content:

from synapseai import crawl_and_query

url = "https://example.com"
query = "What services does this website offer?"

results = crawl_and_query(url, query)

for chunk, similarity in results:
    print(f"Similarity: {similarity:.4f}")
    print(f"Chunk: {chunk[:200]}...")  # Print first 200 characters
    print("-" * 50)

How It Works

• For PDFs: The package extracts text content from the file.
• For Web Pages: It crawls the specified URL and extracts the text content.
• The extracted text is split into manageable chunks.
• The package uses HuggingFace's BGE embeddings to convert text chunks and the query into vector representations.
• Cosine similarity is used to find the most relevant text chunks for the given query.
• The top 5 most relevant chunks are returned along with their similarity scores.

License

This project is licensed under the MIT License - see the LICENSE file for details.