SearchFlow

SearchFlow is an assistant designed to help you index webpages into structured datasets. It leverages various tools and models to scrape, process, and store web content efficiently.

Features

• Web Scraping: Uses trafilatura for focused crawling and web scraping.
• Document Processing: Supports chunking and processing of various document types.
• Database Management: Manages projects, documents, and prompts using PostgreSQL.
• Vector Search: Utilizes vector search for document retrieval.
• LLM Integration: Integrates with language models for question answering and document grading.

Installation

To set up the development environment, use the provided Dockerfile and .devcontainer/devcontainer.json for a consistent development setup.

Prerequisites

• Docker
• Python 3.11 or higher

Steps

Usage

Install SearchFlow via pip:

pip install searchflow

Quickstart

1. Initialize the Database

from searchflow.db.postgresql import DB
db = DB()
db.create_project(project_name="example_project")

2. Create a project

db.create_project(project_name="example_project")

3. Import Data from a URL

from searchflow.importers import WebScraper
scraper = WebScraper(project_name='MyProject', db=db)
scraper.full_import("https://example.com", max_pages=100)

4. ** Upload a file to the project **

from searchflow.importers import Files
with open("path/to/your/file.pdf", "rb") as f:
bytes_data = f.read()
files = Files()
files.upload_file(
document_data=[(bytes_data, "file.pdf")],
project_name="MyProject",
inference_type="local"
)

5. List Files in a Project

files.list_files(project_name="MyProject")

6. Remove a File from a Project

files.remove_file(project_name="MyProject", file_name="file.pdf")

Question Answering

Vector Search

To perform a similarity search:

from searchflow.db.postgresql import DB
db = DB()
results = db.similarity_search(project_name="example_project", query="example query"