html-extract 0.7.0

Extract structured data from HTML files using declarative configuration files (YAML or JSON)

HTML Extract

A Python tool for extracting structured data from HTML files using declarative configuration files (YAML or JSON). No code required - define what to extract, how to find it, and how to process it, all in configuration files. Automatically detects and extracts all items from each HTML file.

Features

• Configuration-Driven: Define extraction rules in YAML or JSON files
• Multiple Extraction Types: Text, regex, contains, and metadata extraction
• Batch Processing: Process single files, directories, or CSV lists
• Multiprocessing: Parallel processing for improved performance
• Flexible Output: CSV and JSON output formats
• Dual Interface: Python API and CLI with complete feature parity
• Programmatic Configs: Create configurations from Python dict objects

Quick Start

Installation

pip install html-extract

This installs the package with all features, including the html-extract CLI command.

Basic Usage

1. Create a configuration file:

html-extract -t config data/config.yaml

2. Customize the configuration to match your HTML structure

3. Extract data:

html-extract data/page.html -c data/config.yaml -k category -o output.csv

Choose Your Path

HTML Extract provides two interfaces - choose the one that fits your workflow:

🖥️ Using the Command Line?

If you prefer working with command-line tools, start here:

→ CLI Quick Start - Get started in 5 minutes

CLI Documentation:

• Installation
• Common Tasks
  • Extract from Single File
  • Process Folder
  • Batch with CSV
  • Generate Templates
• Configuration Guide
• CLI Reference
• Troubleshooting

🐍 Using Python?

If you're building Python applications or scripts, start here:

→ Python Quick Start - Get started in 5 minutes

Python Documentation:

• Installation
• Python API Guide
• Programmatic Configs
• Integration Examples
• API Reference
• Troubleshooting

📚 Shared Resources

Both CLI and Python users can access:

• Configuration Reference - Complete config guide
• Core Concepts - Understanding how it works
• Examples - Real-world use cases
• Logging Guide - Logging configuration

🚀 Advanced Topics

• Performance Optimization
• Custom Processing
• Best Practices

Example

Extract product listings from HTML:

# config.yaml
categories:
  - name: products
    attribute_names: [link, title, price]

attributes:
  - name: link
    required: true
    extract:
      type: regex
      selector: a
      extract_attribute: href
      pattern: "/d/oferta/.*\\.html"
  
  - name: title
    required: true
    extract:
      type: text
      selector: h4
      extract_attribute: text
  
  - name: price
    required: false
    extract:
      type: text
      selector: p
      html_attributes:
        data-testid: ad-price
      extract_attribute: text

Using CLI:

html-extract data/page.html -c config.yaml -k products -o output.csv

Using Python:

from html_extract import extract_data_from_html, load_config, save_output

config = load_config('config.yaml')
df = extract_data_from_html('page.html', config, category='products')
save_output(df, 'output.csv')

Requirements

• Python 3.7+
• beautifulsoup4
• pandas
• pyyaml

Development Status

⚠️ Note: This project was created with vibe coding and is still under heavy development. Features and APIs may change. Use with caution in production environments.

License

This project is licensed under the MIT License - see the LICENSE file for details.