Segment Classifier
An asynchronous Python library that classifies HTML segments extracted by a page-segmenter into structured component types.
Overview
The segment_classifier implements a 4-stage classification pipeline with progressive fallback to optimize for cost and speed:
- Rule-based heuristics — Zero LLM cost. Uses DOM structure, text density, siblings, and attributes.
- L1 exact fingerprint cache — Zero LLM cost. Exact matching on structural DOM fingerprint hashes.
- L2 fuzzy cluster cache — Zero LLM cost. TF-IDF and cosine similarity on fingerprint tokens.
- LLM batch classification — Batched fallback via LiteLLM with feature-based model routing based on segment complexity.
Installation
You can install the package using poetry:
poetry install
Or via pip (once published):
pip install segment-classifier
Setup
The library uses pydantic-settings to manage configuration via a .env file or environment variables.
Required environment variables:
CLASSIFIER_LITELLM_API_KEY="your-api-key"
Usage
import asyncio
from segment_classifier import ClassifierPipeline
from segment_classifier.config import ClassifierSettings
from segment_classifier.models import InputSegment, SegmentPosition
async def main():
settings = ClassifierSettings()
pipeline = ClassifierPipeline(settings)
await pipeline.initialize()
segments = [
InputSegment(
segment_id="seg_001",
page_url="https://example.com/products",
page_slug="products",
raw_html="<div class='product-card'>...</div>",
text_content="Product Item",
position_hint=SegmentPosition.MIDDLE,
sibling_count=3,
)
]
result = await pipeline.run(segments)
await pipeline.shutdown()
for seg in result.classified:
print(seg.component_type)
asyncio.run(main())
Caching
Caches are stored by default in .cache/l1_fingerprints.json and .cache/l2_clusters.json / .cache/l2_embeddings.npy.
Stages Breakdown
Every returned ClassifiedSegment will be marked with a classification_stage indicating which of the 4 stages resolved the query.
Metadata
Release files for segment-classifier 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| segment_classifier-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for segment_classifier-0.3.0-py3-none-any.whl |
Release files / segment_classifier-0.3.0-py3-none-any.whl
| Download URL | segment_classifier-0.3.0-py3-none-any.whl |
|---|---|
| Size | 29.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5bc0c0b5619b6b65b891871c6c90114c4fc3591f31e3386ffcc69ba2e21445bc
|
|
BLAKE2b-256 checksum How to use checksums |
dfde6cdfd8b463ee41c2b0a6c06a42cfdc890f50bcde5d580317a077ec175b99
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.11
|
Security for segment-classifier 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.