Multi-Class Content Ranking using Seq-to-Seq Models

This repository contains code for a multi-class HTML content ranking model using BiLSTM-CRF, as described in the paper "A High-Speed Multi-Class Content Ranking using Seq-to-Seq Models".

Overview

The model classifies HTML content into 6 classes indicating importance:

• Title
• Main content
• Related parts
• Other topics
• Headers and footers
• Useless parts

The model uses only structural HTML features and is language agnostic.

Abstract

Content Ranking is an important task in many applications, such as documents ranking, text retrieval, text summarization and other natural language processing (NLP) procedures. Existing approaches define content ranking as a binary classification boilerplate removal, in which a text is either boilerplate or main-content. While boilerplate removal provides a basic mean for text ranking, in many applications, more importance levels are required to distinguish between different parts of an HTML document.

Based on a new dataset of segmented and labeled HTML documents, in this paper we introduce a high-speed multi-class sequence-to-sequence text classification model for determining the importance level of a text within six levels. As we use only HTML’s structural features in the training process, our model is language-agnostic and also it can be fine-tuned on a small dataset of web pages of other languages.

The involved feature extraction uses an optimized algorithm in terms of speed, which makes the end-to-end inference on an html page in about and less than 25ms, which is a huge benefit in terms of scalability of the model. Besides, our model achieves the 83 percent score in accuracy and 74 percent score in average 6 classes f1 score, which is a prominent result.

Dataset

The dataset used for this project can be found at ZDA Dataset. It contains 1602 labeled HTML documents across 6 classes:

• Title
• Main content
• Related parts
• Other topics
• Headers and footers
• Useless parts

Results

The model achieves 83% accuracy and 74% average F1 score on the test set.

Confusion Matrix

The confusion matrix shows the model confuses classes with adjacent importance levels:

Example Output

The model generalizes well to new HTML pages:

Usage

See the examples for scripts showing end-to-end usage.

Basic usage:

from content_ranker import predict

model = predict.load_model("path/to/model.pt") 

html_string = load_html(...)

features = extract_features(html_string) 

tags = model.predict(features)

Repository Structure

.
├── Dockerfile
├── examples/               - Example scripts showing usage
├── main.py                 - Main entry point  
├── pyproject.toml          - pypi config
├── README.md
├── requirements.txt        - Python requirements
├── src/                    - Source code
│   └── content_ranker/     - Package containing model code
│       ├── features/       - Feature extraction  
│       ├── models/         - PyTorch model definitions
│       ├── resources/      - Resources like pretrained models
│       └── utils/          - Utility functions
└── tests/                  - Tests

Citation

Please cite the paper if you use this repository:

@article{ashourisefat-raziei-2023-content-ranker,
  title={A High-Speed Multi-Class Content Ranking using Seq-to-Seq Models},
  author={AshouriSefat, Soroush and Kazemi, Reza and Alikhany, Morteza and Raziei, Mohammad and Amini, Arash},
  journal={IEEE Access},
  volume={9},
  pages={XXXXX--XXXXX},
  year={2023},
  publisher={IEEE}
}

Contact

For any questions, feel free to open an issue or contact us at soroush_ashourisefat@ee.sharif.edu or mohammadraziei1375@gmail.com .