webpage-content-scraper 0.3

Scrape only the content from a webpage using Firefox's reader view.

Webpage Content Scraper

Scrape only the content from a webpage using Firefox's reader view.

Installation

Install this library using pip:

pip install webpage-content-scraper

Then install the Firefox browser for Playwright:

playwright install firefox

Usage

Simply supply a list of pages and optionally, a format and timeout.

from webpage_content_scraper import fetch_page_content, Formats

urls = [
    'https://microsoft.github.io/autogen/blog/2024/03/03/AutoGen-Update',
    'https://microsoft.github.io/autogen/blog/2024/02/29/StateFlow',
    'https://microsoft.github.io/autogen/blog/2024/02/11/FSM-GroupChat'
]
blog_posts = fetch_page_content(urls, Formats.MARKDOWN)

for post in blog_posts:
    print(f'\n---start---\n{post}\n---end---\n')

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd python-webpage-content-scraper
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest