An assistant helping you to index webpages into structured datasets.
Project description
SearchFlow
SearchFlow is an assistant designed to help you index webpages into structured datasets. It leverages various tools and models to scrape, process, and store web content efficiently.
Features
- Web Scraping: Uses
trafilatura
for focused crawling and web scraping. - Document Processing: Supports chunking and processing of various document types.
- Database Management: Manages projects, documents, and prompts using PostgreSQL.
- Vector Search: Utilizes vector search for document retrieval.
- LLM Integration: Integrates with language models for question answering and document grading.
Installation
To set up the development environment, use the provided Dockerfile
and .devcontainer/devcontainer.json
for a consistent development setup.
Prerequisites
- Docker
- Python 3.11 or higher
Steps
Usage
Install SearchFlow via pip:
pip install searchflow
Quickstart
- Initialize the Database
from searchflow.db.postgresql import DB
db = DB()
db.create_project(project_name="example_project")
- Create a project
db.create_project(project_name="example_project")
- Import Data from a URL
from searchflow.importers import WebScraper
scraper = WebScraper(project_name='MyProject', db=db)
scraper.full_import("https://example.com", max_pages=100)
- ** Upload a file to the project **
from searchflow.importers import Files
with open("path/to/your/file.pdf", "rb") as f:
bytes_data = f.read()
files = Files()
files.upload_file(
document_data=[(bytes_data, "file.pdf")],
project_name="MyProject",
inference_type="local"
)
- List Files in a Project
files.list_files(project_name="MyProject")
- Remove a File from a Project
files.remove_file(project_name="MyProject", file_name="file.pdf")
Question Answering
Vector Search
To perform a similarity search:
from searchflow.db.postgresql import DB
db = DB()
results = db.similarity_search(project_name="example_project", query="example query"
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
searchflow-0.0.89.tar.gz
(33.3 kB
view hashes)
Built Distribution
Close
Hashes for searchflow-0.0.89-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 334983af978566b283db7c0328044a9d824f42669c719ca4a3c4596ed5160eb8 |
|
MD5 | 21968c0473a5cdaa6b1c005bdc5921be |
|
BLAKE2b-256 | 6b0ede061ada518415a7172f6164ef98d576e61f8bb09f919d75fee94b49d8a7 |