Skip to main content

py-web-text-extractor

CI PyPI version Python 3.14+ License: MIT

A CLI tool and Python library to extract clean text content from web pages.

Overview

py-web-text-extractor provides a simple interface for extracting the main text content from HTML documents. It can be used as a command-line tool for quick extractions or as a Python library for integration into other applications. The tool uses a fallback strategy, trying markitdown first and then trafilatura to ensure high reliability.

Features

  • Dual Extractor Strategy: Uses markitdown as the primary extractor and falls back to trafilatura for robustness.
  • CLI and Library Interface: Can be used as a standalone command-line tool or as a Python library.
  • Error Handling Modes: Supports a strict mode that raises exceptions on failure and a safe mode that returns an empty string.
  • Modern Python: Fully typed with Python 3.14+ support.

Prerequisites

  • Python 3.14 or higher.

Installation

You can install the package using pip or uv.

Using pip

pip install py-web-text-extractor

Using uv

uv add py-web-text-extractor

Usage

The tool can be used via its command-line interface or as a Python library.

Command-Line Interface (CLI)

The CLI is the quickest way to extract text from a URL.

Basic Extraction:

py-web-text-extractor https://example.com

Safe Mode:

In safe mode, the tool will return an empty string and exit gracefully if an error occurs.

py-web-text-extractor https://example.com --safe

Verbose Mode:

For troubleshooting, verbose mode provides detailed debug output.

py-web-text-extractor https://example.com --verbose

CLI Exit Codes:

Code Meaning
0 Success
1 No text content found
2 Invalid URL
3 Text extraction failed
4 Unexpected error

Python Library

For programmatic use, you can import the ExtractorService.

Quick Start:

from py_web_text_extractor.service.extractor_service import ExtractorService
from py_web_text_extractor.exception.exceptions import TextExtractionError, UrlIsNotValidException

# Initialize the service
service = ExtractorService()

# Strict mode: raises an exception on failure
try:
    text = service.extract_text_from_page("https://example.com")
    print(text)
except UrlIsNotValidException:
    print("The provided URL is not valid.")
except TextExtractionError as e:
    print(f"Failed to extract text: {e}")

# Safe mode: returns an empty string on failure
text_safe = service.extract_text_from_page_safe("https://invalid-url")
if not text_safe:
    print("Extraction failed or no content found.")

API Reference

ExtractorService

The main class for handling text extraction.

Methods:

  • extract_text_from_page(url: str) -> str: Extracts text from the given URL. Raises a TextExtractionError or UrlIsNotValidException on failure.
  • extract_text_from_page_safe(url: str) -> str: Extracts text from the given URL. Returns an empty string on failure.

Exceptions

The library uses a set of custom exceptions to allow for specific error handling.

  • TextExtractionError: Base exception for the library.
  • UrlIsNotValidException: Raised for invalid URL formats.
  • TextExtractionFailure: Raised when all extraction attempts fail.
  • MarkItDownExtractionException: Specific failure from the markitdown extractor.
  • TrafilaturaExtractionException: Specific failure from the trafilatura extractor.

Architecture

The service employs a fallback strategy to maximize reliability:

  1. It first attempts to extract content using markitdown.
  2. If markitdown fails (e.g., returns a blank string or raises an error), the service automatically retries the extraction using trafilatura.
  3. The first successful result is returned. If both extractors fail, an error is raised or an empty string is returned, depending on the mode.

Testing

To run the test suite, first install the development dependencies and then run pytest.

Patch pydub for Python 3.14 compatibility For Python 3.14 can be required patching of old dependency

python scripts/patch_pydub.py
# Install dev dependencies
uv pip install -e ".[dev]"

# Run tests
uv run pytest

# Run linters and type checkers
uv run ruff check .
uv run ruff format .
uv run mypy src/

License

This project is licensed under the MIT License. See the LICENSE file for details.

Metadata

Release files for py-web-text-extractor 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for py-web-text-extractor 0.1.0
File Size Uploaded Provenance Actions
py_web_text_extractor-0.1.0.tar.gz 8.3 kB Without provenance Details for py_web_text_extractor-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for py-web-text-extractor 0.1.0
File Interpreter ABI Platform Actions
py_web_text_extractor-0.1.0-py3-none-any.whl Python 3 none any Details for py_web_text_extractor-0.1.0-py3-none-any.whl

Total release size: 22.1 kB

Release files / py_web_text_extractor-0.1.0.tar.gz

Download URL py_web_text_extractor-0.1.0.tar.gz
Size 8.3 kB
Tags Source
SHA-256 checksum
How to use checksums
26bb31457cf284da831d6fc279076b80e3ee9cbf28c949aaa2bce1c0cbcf68db
BLAKE2b-256 checksum
How to use checksums
50c5e1ceb8f4e2377d8cd8b8b506c320441a9189fb0e7459acd2d7834f3643e6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / py_web_text_extractor-0.1.0-py3-none-any.whl

Download URL py_web_text_extractor-0.1.0-py3-none-any.whl
Size 13.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5638c94c98cdfaa5e3e5d1395ee842f54426b0b18889eef9438010afbc994bc1
BLAKE2b-256 checksum
How to use checksums
dd198beb710e7c4f5d09cd2a31f0e8c04ca359b54eab4533c0e0dcf7000a761c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for py-web-text-extractor 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page