py-web-text-extractor
A CLI tool and Python library to extract clean text content from web pages.
Overview
py-web-text-extractor provides a simple interface for extracting the main text content from HTML documents. It can be used as a command-line tool for quick extractions or as a Python library for integration into other applications. The tool uses a fallback strategy, trying markitdown first and then trafilatura to ensure high reliability.
Features
- Dual Extractor Strategy: Uses
markitdownas the primary extractor and falls back totrafilaturafor robustness. - CLI and Library Interface: Can be used as a standalone command-line tool or as a Python library.
- Error Handling Modes: Supports a strict mode that raises exceptions on failure and a safe mode that returns an empty string.
- Modern Python: Fully typed with Python 3.14+ support.
Prerequisites
- Python 3.14 or higher.
Installation
You can install the package using pip or uv.
Using pip
pip install py-web-text-extractor
Using uv
uv add py-web-text-extractor
Usage
The tool can be used via its command-line interface or as a Python library.
Command-Line Interface (CLI)
The CLI is the quickest way to extract text from a URL.
Basic Extraction:
py-web-text-extractor https://example.com
Safe Mode:
In safe mode, the tool will return an empty string and exit gracefully if an error occurs.
py-web-text-extractor https://example.com --safe
Verbose Mode:
For troubleshooting, verbose mode provides detailed debug output.
py-web-text-extractor https://example.com --verbose
CLI Exit Codes:
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | No text content found |
| 2 | Invalid URL |
| 3 | Text extraction failed |
| 4 | Unexpected error |
Python Library
For programmatic use, you can import the ExtractorService.
Quick Start:
from py_web_text_extractor.service.extractor_service import ExtractorService
from py_web_text_extractor.exception.exceptions import TextExtractionError, UrlIsNotValidException
# Initialize the service
service = ExtractorService()
# Strict mode: raises an exception on failure
try:
text = service.extract_text_from_page("https://example.com")
print(text)
except UrlIsNotValidException:
print("The provided URL is not valid.")
except TextExtractionError as e:
print(f"Failed to extract text: {e}")
# Safe mode: returns an empty string on failure
text_safe = service.extract_text_from_page_safe("https://invalid-url")
if not text_safe:
print("Extraction failed or no content found.")
API Reference
ExtractorService
The main class for handling text extraction.
Methods:
extract_text_from_page(url: str) -> str: Extracts text from the given URL. Raises aTextExtractionErrororUrlIsNotValidExceptionon failure.extract_text_from_page_safe(url: str) -> str: Extracts text from the given URL. Returns an empty string on failure.
Exceptions
The library uses a set of custom exceptions to allow for specific error handling.
TextExtractionError: Base exception for the library.UrlIsNotValidException: Raised for invalid URL formats.TextExtractionFailure: Raised when all extraction attempts fail.MarkItDownExtractionException: Specific failure from themarkitdownextractor.TrafilaturaExtractionException: Specific failure from thetrafilaturaextractor.
Architecture
The service employs a fallback strategy to maximize reliability:
- It first attempts to extract content using
markitdown. - If
markitdownfails (e.g., returns a blank string or raises an error), the service automatically retries the extraction usingtrafilatura. - The first successful result is returned. If both extractors fail, an error is raised or an empty string is returned, depending on the mode.
Testing
To run the test suite, first install the development dependencies and then run pytest.
Patch pydub for Python 3.14 compatibility For Python 3.14 can be required patching of old dependency
python scripts/patch_pydub.py
# Install dev dependencies
uv pip install -e ".[dev]"
# Run tests
uv run pytest
# Run linters and type checkers
uv run ruff check .
uv run ruff format .
uv run mypy src/
License
This project is licensed under the MIT License. See the LICENSE file for details.
Metadata
Release files for py-web-text-extractor 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| py_web_text_extractor-0.1.0.tar.gz | 8.3 kB | Without provenance | Details for py_web_text_extractor-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| py_web_text_extractor-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for py_web_text_extractor-0.1.0-py3-none-any.whl |
Total release size: 22.1 kB
Release files / py_web_text_extractor-0.1.0.tar.gz
| Download URL | py_web_text_extractor-0.1.0.tar.gz |
|---|---|
| Size | 8.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
26bb31457cf284da831d6fc279076b80e3ee9cbf28c949aaa2bce1c0cbcf68db
|
|
BLAKE2b-256 checksum How to use checksums |
50c5e1ceb8f4e2377d8cd8b8b506c320441a9189fb0e7459acd2d7834f3643e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / py_web_text_extractor-0.1.0-py3-none-any.whl
| Download URL | py_web_text_extractor-0.1.0-py3-none-any.whl |
|---|---|
| Size | 13.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5638c94c98cdfaa5e3e5d1395ee842f54426b0b18889eef9438010afbc994bc1
|
|
BLAKE2b-256 checksum How to use checksums |
dd198beb710e7c4f5d09cd2a31f0e8c04ca359b54eab4533c0e0dcf7000a761c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for py-web-text-extractor 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.