Skip to main content

An assistant helping you to index webpages into structured datasets.

Project description

SearchFlow

SearchFlow is an assistant designed to help you index webpages into structured datasets. It leverages various tools and models to scrape, process, and store web content efficiently.

Features

  • Web Scraping: Uses trafilatura for focused crawling and web scraping.
  • Document Processing: Supports chunking and processing of various document types.
  • Database Management: Manages projects, documents, and prompts using PostgreSQL.
  • Vector Search: Utilizes vector search for document retrieval.
  • LLM Integration: Integrates with language models for question answering and document grading.

Installation

To set up the development environment, use the provided Dockerfile and .devcontainer/devcontainer.json for a consistent development setup.

Prerequisites

  • Docker
  • Python 3.11 or higher

Steps

  1. Clone the repository:

    git clone https://github.com/yourusername/searchflow.git
    cd searchflow
    
  2. Build the Docker container:

    docker build -t searchflow .
    
  3. Run the Docker container:

    docker run -it -p 8501:8501 searchflow
    

Usage

Web Scraping

To scrape a website and index the links:

from searchflow.importers.webscraper import WebScraper
scraper = WebScraper(project_name="example_project")
scraper.get_all_links(base_url="https://example.com")

Document Processing

To upload and process files:

from searchflow.importers.file_importer import FileImporter
files = Files()
files.upload_file(document_data=[(b"file_content", "example.pdf")], project_name="example_project")

Vector Search

To perform a similarity search:

from searchflow.db.postgresql import DB
db = DB()
results = db.similarity_search(project_name="example_project", query="example query"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

searchflow-0.0.78.tar.gz (28.3 kB view hashes)

Uploaded Source

Built Distribution

searchflow-0.0.78-py3-none-any.whl (36.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page