Page Classifier

An asynchronous webpage type classifier that uses URL, metadata, structural, and content signals to accurately determine the type of a given web page (e.g., product page, blog post, contact-us, etc.).

Installation

You can install the package directly from PyPI (once published):

pip install page-classifier

For development, clone the repository and install it using the Makefile or pip:

git clone <your-repository-url>
cd page-classifier
make install
# or for dev dependencies
make dev

Usage

Python API

You can use the classifier programmatically in your asynchronous Python applications:

import asyncio
from page_classifier import PageClassifier, ClassifierConfig

async def main():
    # Initialize the classifier
    classifier = PageClassifier(config=ClassifierConfig())
    
    # Classify a URL
    result = await classifier.classify_url(
        url="https://www.ganpatihandicrafts.com/printed-kurti.html"
    )
    
    # View the results
    print(result.to_dict())

if __name__ == "__main__":
    asyncio.run(main())

Command-Line Interface (CLI)

The package provides a built-in CLI to easily classify pages from your terminal:

python -m page_classifier "https://www.ganpatihandicrafts.com/printed-kurti.html"

CLI Options

• url: The URL to fetch and classify.
• --platform NAME: Force a specific platform instead of auto-detecting. Only that platform's routing rules will fire.
• --timeout SECONDS: HTTP fetch timeout (default: 15.0).
• --json: Print the full result as JSON instead of a summary.
• --list-platforms: List all supported platform names and exit.

Example with JSON output and timeout:

python -m page_classifier "https://example.com/product/123" --timeout 10 --json

Development

A Makefile is included to streamline development tasks:

• make install: Install the project.
• make dev: Install the project with development dependencies.
• make test: Run the pytest suite.
• make build: Build the distribution packages (sdist and wheel).
• make publish: Build and publish the package to PyPI using twine.
• make clean: Clean up build artifacts and cache directories.