pyprocessors_metadata_from_categories

Turn the categories carried by a document into a metadata value.

How it works

For every input document the processor converts its categories into a metadata entry and then clears the categories. Conversion happens systematically at both levels:

• Document level — the document's categories are turned into document.metadata[metadata_name], then document.categories is cleared.
• Segment level — the same conversion is applied to each sentence: its categories become sentence.metadata[metadata_name], then the sentence's categories are cleared.

For a given set of categories, only those whose score is greater than or equal to multi_label_threshold are kept, and the resulting metadata value is:

• None (the metadata key is removed) when no category passes the threshold,
• a single value when exactly one category passes,
• a list of values when several categories pass.

Parameters

Installation

You can simply pip install pyprocessors_metadata_from_categories.

Developing

Pre-requesites

You will need to install uv (for managing the virtual environment and running tests):

pip install uv

Clone the repository:

git clone https://github.com/oterrier/pyprocessors_metadata_from_categories

Running the test suite

You can run the full test suite with:

uv run pytest

Linting

uv run ruff check .
uv run ruff format --check .

Building the documentation

You can build the HTML documentation with:

uv run --extra docs sphinx-build docs docs/_build

The built documentation is available at docs/_build/index.html.

SBOM & vulnerability check

Install the SBOM dependencies:

uv sync --extra sbom

Generate a CycloneDX SBOM from the current environment:

uv run cyclonedx-py environment -o sbom.cdx.json --output-format json

Audit dependencies for known vulnerabilities:

uv run pip-audit --format json --output audit-report.json

To fail on any known vulnerability (useful in CI):

uv run pip-audit --strict