pysegmenters-syntok 1.8.54

syntok segmenter

pysegmenters_syntok

Rule based segmenter based on syntok

Installation

You can simply pip install pysegmenters_syntok.

Developing

Pre-requesites

You will need to install uv (for building and managing the package):

pip install uv

Clone the repository:

git clone https://github.com/oterrier/pysegmenters_syntok

Running the test suite

You can run the full test suite against Python 3.12 with:

uv run pytest

Linting

uv run ruff check .
uv run ruff format --check .

Building the documentation

You can build the HTML documentation with:

uv run --extra docs sphinx-build docs docs/_build

The built documentation is available at docs/_build/index.html.

SBOM & vulnerability check

Install the SBOM dependencies:

uv sync --extra sbom

Generate a CycloneDX SBOM from the current environment:

uv run cyclonedx-py environment -o sbom.cdx.json --output-format json

Audit dependencies for known vulnerabilities:

uv run pip-audit --format json --output audit-report.json

To fail on any known vulnerability (useful in CI):

uv run pip-audit --strict