webpage-content-scraper 0.1

Scrape only the content from a webpage using Firefox's reader view.

Webpage Content Scraper

Scrape only the content from a webpage using Firefox's reader view.

Installation

Install this library using pip:

pip install webpage-content-scraper

Then install the Firefox browser for Playwright:

playwright install firefox

Usage

Simply supply a URL and optionally, a format and timeout.

from webpage_content_scraper import fetch_page_content, Formats

url = 'https://microsoft.github.io/autogen/blog/2023/10/18/RetrieveChat/'

html_content = fetch_page_content(url)
markdown_content = fetch_page_content(url, Formats.MARKDOWN)

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd python-webpage-content-scraper
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest