webdata-extractor 1.0.0

Pipeline-based web content extraction library

WebData Extractor

Pipeline-based web content extraction library for extracting company information and metadata from web pages.

🚀 Features

• Pipeline Architecture: Modular architecture with independent processors
• Extensibility: Easy to add new processors without modifying existing code
• Auto-Discovery: Automatic discovery of processors in the folder
• Prioritization: Control processor execution order
• Conditional Execution: Processors can execute conditionally
• Error Handling: Pipeline continues working even if individual processors fail

📦 Installation

pip install -e .

Or for development:

pip install -e ".[dev]"

🔧 Usage

Simple Usage

from web_extractor import extract_company_info

# Extract company information
result = extract_company_info("example.com")

print(result["domain"])                    # Domain
print(result["metadata"]["og:site_name"]) # Company name (if available)
print(result["metadata"])                  # All metadata

With Processor Selection

from web_extractor import extract_company_info

# Use only standard processors
result = extract_company_info("example.com", use_all_processors=False)

# Or automatically discover all processors (default)
result = extract_company_info("example.com", use_all_processors=True)

Advanced Usage

from web_extractor import fetch_url, Pipeline
from web_extractor.processors import (
    MetaTagsProcessor,
    TitleProcessor,
    CustomProcessor  # Your custom processor
)

# Fetch HTML
data = fetch_url("example.com")

# Configure pipeline
pipeline = Pipeline()
pipeline.add_processors([
    MetaTagsProcessor(),
    TitleProcessor(),
    CustomProcessor(),
])

# Execute processing
result = pipeline.execute(data, data["soup"], data["html"])

Auto-Discovery

from web_extractor import discover_processors, extract_with_auto_discovery

# Show all discovered processors
processors = discover_processors()
for proc in sorted(processors, key=lambda p: p.priority):
    print(f"[{proc.priority:3d}] {proc.name}")

# Use all discovered processors
result = extract_with_auto_discovery("example.com")

🔌 Creating a Custom Processor

1. Create a file in web_extractor/processors/my_processor.py:

from typing import Any, Dict
from bs4 import BeautifulSoup
from web_extractor.pipeline import DataProcessor


class MyProcessor(DataProcessor):
    @property
    def name(self) -> str:
        return "my_processor"
    
    @property
    def priority(self) -> int:
        return 60  # Execution priority
    
    def process(self, data: Dict[str, Any], soup: BeautifulSoup, html: str) -> Dict[str, Any]:
        # Your processing logic
        meta_tag = soup.find("meta", attrs={"name": "custom-tag"})
        if meta_tag:
            content = meta_tag.get("content")
            if content:
                data["metadata"]["custom_data"] = str(content)
        
        return data

2. Add to web_extractor/processors/__init__.py:

from .my_processor import MyProcessor

__all__ = [
    # ... others
    'MyProcessor',
]

3. Done! The processor will execute automatically.

📊 Result Structure

{
    "url": "https://example.com",
    "domain": "example.com",
    "metadata": {
        "og:site_name": "Example Company",
        "og:title": "Example - Home",
        "og:description": "...",
        "twitter:site": "examplecom",
        "title_raw": "Example Company - Official Site",
        "title": "Example Company",
        "description": "...",
        "copyright": {...},
        # ... other metadata
    },
    "processors_executed": [
        "meta_tags",
        "title",
        "copyright",
        "domain_fallback"
    ]
}

🎯 Existing Processors

Priority	Processor	Description
10	meta_tags	All meta tags (OpenGraph, Twitter, Dublin Core, Apple, Microsoft, Facebook, Article, etc.)
30	title	HTML title, application-name, title variants, title parts
40	copyright	Copyright from text, meta tags, schema.org, footer, link rel
999	domain_fallback	Fallback to domain name

📝 Examples

See the examples/ folder for complete examples:

• examples/basic_usage.py - Basic usage
• examples/custom_processor.py - Creating a custom processor
• examples/advanced_pipeline.py - Advanced pipeline configuration

🧪 Testing

# Run tests
pytest

# With coverage
pytest --cov=web_extractor --cov-report=html

📄 License

MIT License

🤝 Contributing

Contributions are welcome! Please create an issue or pull request.