htms 0.1.10

A Python web scraping library using HTML

HTMS: HTML-based Web Scraping Library

Overview

HTMSLParser is a Python library for web scraping using an HTML-like configuration. It simplifies data extraction, handles pagination, and supports custom parsing, making it easy to gather structured data from websites with minimal coding.

Features

• Declarative Scraping Configuration: Define scraping tasks using an HTML-like syntax to specify requests, lists, items, and outputs.
• Supports XPath Data Extraction
• Pagination Handling: Automatically manage pagination by defining start, end, and URL templates.
• Follow-up Requests: Enable follow-up scraping on specific items based on extracted data from initial requests.
• Multiple Parsers: Combine and use multiple parsers for complex scraping tasks.
• Flexible Data Extraction: Use XPath for precise data extraction, including support for custom parsing and filtering.
• JSON Output: Export scraped data directly to JSON files.
• Item Data Manipulation: Strip whitespace, parse data, and apply filters to extracted items.
• Request Configuration: Supports GET and POST methods with customizable headers.

Installation

You can install the package using pip if it is available on PyPI:

pip install htms

Alternatively, to install htms, simply clone the repository and include it in your project:

git clone https://github.com/BowangLan/htms.git

Getting Started

htms lets you specify common web scraping tasks in HTML. To run an htms HTML file therefore is to run web scraping tasks described inside the HTML file.

Command for running an htms HTML file:

python -m htms <your-html-file>

Alternatively, you can use call the Python API to run the HTML file:

from htms import HTMSParser

with open('your-html-file.html', 'r') as file:
    html_str = file.read()

parser = HTMSParser()
parser.feed(html_str)
parser.start()

Run Examples

A list of example HTML files are included in this repo under src/htms/example_html folder. To run these, use this command:

python -m htms.examples <example-file-name>

You can see a full list of example files using this command:

python -m htms.examples

HTML Configuration

Basic Example

Suppose you want to scrape badminton club locations from badmintonclubs.org. The following configuration will extract the name and href of each club listed on the site:

<request url="https://badmintonclubs.org">
  <list xpath="//map/area" name="locations" key="href">
    <item
      name="name"
      xpath="@title"
      parse="value.split('badminton in ')[1] if value else None"
    ></item>
    <item name="href" xpath="@href"></item>
  </list>
</request>

You make a request by a RequestTag . Then, you create a ListTag inside this request tag for parsing the output of this request. Then, for each location element extract by xpath in the ListTag , you extract two fields (name and value) by nesting two ItemTag inside the ListTag .

• key attribute in the list tag tells the parser to remove duplicates of the extracted object list by the provided key. In the above case, locations that has the same href are dropped.

This HTML snippet is equivalent to the followinig Python code:

import requests
from lxml import html

output = []

url = "https://badmintonclubs.org"
response = requests.get(url)

tree = html.fromstring(response.content)

locations = tree.xpath("//map/area")

result = []
for location in locations:
    name = location.xpath("@title")[0] if location.xpath("@title") else None
    href = location.xpath("@href")[0] if location.xpath("@href") else None
    
    # Parse the name field according to the provided logic
    if name:
        name = name.split('badminton in ')[1] if 'badminton in ' in name else None
    
    result.append({
        "name": name,
        "href": href
    })

href_set = set()
new_result = []
for item in result:
    if item['href'] in href_set:
        continue
    href_set.add(item['href'])
    new_result.append(iem)
result = new_result

output.append({
  "locations": result
})

ListTag and ItemTag

ItemTag and ListTag are abstractions designed to simplify the process of extracting and processing data from HTML responses in a structured and intuitive manner. These abstractions allow you to define the extraction logic using a declarative syntax, making it easy to manage complex scraping tasks.

ItemTag

The ItemTag represents a single data extraction operation. It is used to extract a single value from a given HTML element based on the specified XPath.

Attributes for both ItemTag and ListTag :

• name: The name of the field being extracted.
• id (optional): An optional global id of the parser.
• xpath: The XPath expression used to locate the data within the HTML element.
• parse (optional): An optional inline Python expression runs after all other processing steps and right before returning the extracted value.
• pre-parse (optional): An optional inline Python expression that runs before passing the value to child parsers.

Attributes specific to ItemTag :

• strip: Strip whitespace and \n from string values.

Example Usage:

<item name="title" xpath="//h1/text()" parse="value.strip()"></item>

This example extracts the text content of an <h1> element, and then applies a strip() operation to remove any leading or trailing whitespace.

ListTag

The ListTag is an extension of ItemTag that handles the extraction of multiple values from a list of HTML elements. It is useful when you need to process a collection of similar elements, such as a list of articles or links.

Attributes specific to ListTag :

• All attributes of ItemTag.
• key (optional): The field by which to remove duplicates from the list.
• filter (optional)

Example Usage:

<list xpath="//div[@class='articles']/article" name="articles" key="url">
  <item name="title" xpath=".//h2/text()" />
  <item name="url" xpath=".//a/@href" />
</list>

This example extracts a list of articles from a page. Each article has a title and a url, and duplicates are removed based on the url field.

RequestTag

The RequestTag represents a single HTTP request and the associated data extraction process.

The response of the request is parsed based on the type attribute, which specifies the type of response.

The child tags of a RequestTag can be:

• ItemTag or ListTag for data extraction.
• ExportTag for exporting data.
• ... (TODO)

Attributes:

• url: URL of the request.
• parsers (optional): An optional string field for specifying external parsers ( ListTag or ItemTag ) by id separated by , .
• type (optional): Specifies the type of response. Either "json" or "html". Default to "html" .
• method (optional)