RAGScraper is a Python library designed for efficient and intelligent scraping of web documentation and content. Tailored for Retrieval-Augmented Generation systems, RAGScraper extracts and preprocesses text into structured, machine-learning-ready formats. It emphasizes precision, context preservation, and ease of integration with RAG models, making it an ideal tool for developers looking to enhance AI-driven applications with rich, web-sourced knowledge.
Project description
RAGScraper
RAGScraper is a simple Python package that scrapes webpages and converts them to markdown format for RAG usage.
Installation
To install RAGScraper, simply run:
pip install ragscraper
Usage
To use RAGScraper as a command-line tool:
rag-scraper <URL>
To use RAGScraper in a Python script:
from rag_scraper.scraper import Scraper
from rag_scraper.converter import Converter
# Fetch HTML content
url = "https://example.com"
html_content = Scraper.fetch_html(url)
# Convert to Markdown
markdown_content = Converter.html_to_markdown(
html=html_content,
base_url=base_url,
parser_features='html.parser',
ignore_links=True
)
print(markdown_content)
Development
To run the tests for RAGScraper, navigate to the package directory and run:
python -m unittest discover tests
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ragscraper-11.5.2023-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1c4553ea6b80634015d7674aaeeb3aff50d440abe604e10973d267f19bd5466 |
|
MD5 | 6241fa19c3740baaa6e620eaa8b20bdf |
|
BLAKE2b-256 | 5a0f86f9e2aa26551b6e86d28892170f5480c3fef36e107e855667afd15e639e |