atai-web-tool 0.0.9

Extract the main content from a webpage using Playwright, readability-lxml, and BeautifulSoup.

atai-web-tool

atai-web-tool is a command-line utility that extracts the main content from a webpage. It leverages zendriver, readability-lxml, and BeautifulSoup to fetch pages, extract primary content, and display a clean, text-only version.

Features

• Headless Browsing: Fetch webpages using zendriver.
• Content Extraction: Extract main content with readability-lxml.
• Clean Output: Remove unwanted HTML tags using BeautifulSoup.
• Easy CLI: Run from the terminal with a single command.

Installation

You can install atai-web-tool via pip:

pip install atai-web-tool

If you prefer to install from source, clone the repository and run:

pip install .

Usage

Extract the main content from a webpage by running:

atai-web-tool https://example.com

This command will open the specified URL, extract the primary content, and print it to the terminal.

Requirements

• Python 3.6 or higher
• zendriver
• readability-lxml
• BeautifulSoup4
• lxml[html_clean]

Development

For local development, install the required dependencies using:

pip install -r requirements.txt

Contributing

Contributions are welcome! Please fork the repository and submit a pull request with your improvements. For major changes, open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License. See the LICENSE file for details.