A Python package for detecting irrelevant content in text and HTML.
Project description
Irrelevant Content Detection
Irrelevant Content Detection is a Python package for detecting and cleaning irrelevant content from text and HTML. It leverages machine learning techniques such as TF-IDF and KMeans clustering to identify and remove non-relevant information from documents.
Table of Contents
Installation
You can install the package using pip:
pip install irrelevant-content-detection
Alternatively, you can clone the repository and install it locally:
git clone https://github.com/berkbirkan/irrelevant-content-detection.git
cd irrelevant-content-detection
pip install .
Usage
The package provides several functions to detect and clean irrelevant content from text and HTML.
Calculate Relevance Scores
The calculate_relevance_scores function calculates the TF-IDF scores for a list of texts.
from irrelevant_content_detection import calculate_relevance_scores
texts = [
"Python is a programming language.",
"This text is not relevant."
]
tfidf_scores = calculate_relevance_scores(texts)
print(tfidf_scores)
Detect Irrelevant Content in Text
The detect_irrelevant_contents function detects irrelevant content from a list of texts.
from irrelevant_content_detection import detect_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
irrelevant_texts = detect_irrelevant_contents(texts)
print(irrelevant_texts)
Clean Irrelevant Content from Text
The clean_irrelevant_contents function removes irrelevant content from a list of texts.
from irrelevant_content_detection import clean_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
cleaned_texts = clean_irrelevant_contents(texts)
print(cleaned_texts)
Extract Text from HTML
The extract_text_from_html function extracts all text from an HTML string.
from irrelevant_content_detection import extract_text_from_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>This text is not relevant.</p>
</body>
</html>
"""
texts = extract_text_from_html(html)
print(texts)
Detect Irrelevant Content in HTML
The detect_irrelevant_html function detects irrelevant content from an HTML string.
from irrelevant_content_detection import detect_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
irrelevant_html = detect_irrelevant_html(html)
print(irrelevant_html)
Clean Irrelevant Content from HTML
The clean_irrelevant_html function removes irrelevant content from an HTML string.
from irrelevant_content_detection import clean_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
cleaned_html = clean_irrelevant_html(html)
print(cleaned_html)
Testing
To run the tests, you can use unittest which is included in the Python Standard Library:
python -m unittest discover
Or you can run the test file directly:
python test_detector.py
Contributing
Contributions are welcome! Please follow these steps to contribute:
- Fork the repository.
- Create a new branch with your feature or bugfix.
- Commit your changes.
- Push to your branch.
- Create a pull request.
License
This project is licensed under the MIT License. See the LICENSE file for more details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file irrelevant_content_detection-0.2.tar.gz.
File metadata
- Download URL: irrelevant_content_detection-0.2.tar.gz
- Upload date:
- Size: 4.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.11.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a739cc07d9511f95a1aa5282f4415950931e89b6353ae2a1c5bfddd26fe7b96f
|
|
| MD5 |
1e58e9eb0eccf883cc9dea1af60a3ba5
|
|
| BLAKE2b-256 |
b5f4256d43b182a0157e6eab8d8a4a8958517167c6c0eb90c2913fd3bfed8cf0
|
File details
Details for the file irrelevant_content_detection-0.2-py3-none-any.whl.
File metadata
- Download URL: irrelevant_content_detection-0.2-py3-none-any.whl
- Upload date:
- Size: 3.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.11.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d6ae6be130ef516afca693fbbc05993bc9d0e6d6f75274fbfe095eb0a14e22d3
|
|
| MD5 |
a190f54cf300b12d92841fffc987aff5
|
|
| BLAKE2b-256 |
8e1e81168ab9bf019e61fd63ee66e35a21c5d30cf3aae542ad4e243b78f5cdd4
|
