Skip to main content

Open in Dev Containers Open in GitHub Codespaces

✂️ wtpsplit-lite

🪓 wtpsplit is a Python package that offers training, inference, and evaluation of state-of-the-art Segment any Text (SaT) models for partitioning text into sentences.

✂️ wtpsplit-lite is a lightweight version of wtsplit that only retains accelerated ONNX inference of SaT models with minimal dependencies:

  1. huggingface-hub to download the model
  2. numpy to process the model in- and output
  3. onnxruntime to run the model
  4. tokenizers to tokenize the text for the model

Installing

To install this package, run:

pip install wtpsplit-lite

Using

Example usage:

from wtpsplit_lite import SaT

text = """
It is known that Maxwell’s electrodynamics—as usually understood at the
present time—when applied to moving bodies, leads to asymmetries which do
not appear to be inherent in the phenomena. Take, for example, the recipro-
cal electrodynamic action of a magnet and a conductor.
"""

# Fast (~150ms/page), good quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256)

# Slow, highest quality:
sat = SaT("sat-12l-sm")
sentences = sat.split(text)

This package also contributes a new 'hat' weighting scheme to wtpsplit that improves output quality when using large strides. To enable it, set weighting="hat" as follows:

# Fast (~150ms/page), better quality:
sat = SaT("sat-3l-sm")
sentences = sat.split(text, stride=128, block_size=256, weighting="hat")

Contributing

Prerequisites
  1. Generate an SSH key and add the SSH key to your GitHub account.

  2. Configure SSH to automatically load your SSH keys:

    cat << EOF >> ~/.ssh/config
    
    Host *
      AddKeysToAgent yes
      IgnoreUnknown UseKeychain
      UseKeychain yes
      ForwardAgent yes
    EOF
    
  3. Install Docker Desktop.

  4. Install VS Code and VS Code's Dev Containers extension. Alternatively, install PyCharm.

  5. Optional: install a Nerd Font such as FiraCode Nerd Font and configure VS Code or PyCharm to use it.

Development environments

The following development environments are supported:

  1. ⭐️ GitHub Codespaces: click on Open in GitHub Codespaces to start developing in your browser.

  2. ⭐️ VS Code Dev Container (with container volume): click on Open in Dev Containers to clone this repository in a container volume and create a Dev Container with VS Code.

  3. ⭐️ uv: clone this repository and run the following from root of the repository:

    # Create and install a virtual environment
    uv sync --python 3.10 --all-extras
    
    # Activate the virtual environment
    source .venv/bin/activate
    
    # Install the pre-commit hooks
    pre-commit install --install-hooks
    
  4. VS Code Dev Container: clone this repository, open it with VS Code, and run Ctrl/⌘ + ⇧ + P → Dev Containers: Reopen in Container.

  5. PyCharm Dev Container: clone this repository, open it with PyCharm, create a Dev Container with Mount Sources, and configure an existing Python interpreter at /opt/venv/bin/python.

Developing
  • This project follows the Conventional Commits standard to automate Semantic Versioning and Keep A Changelog with Commitizen.
  • Run poe from within the development environment to print a list of Poe the Poet tasks available to run on this project.
  • Run uv add {package} from within the development environment to install a run time dependency and add it to pyproject.toml and uv.lock. Add --dev to install a development dependency.
  • Run uv sync --upgrade from within the development environment to upgrade all dependencies to the latest versions allowed by pyproject.toml. Add --only-dev to upgrade the development dependencies only.
  • Run cz bump to bump the package's version, update the CHANGELOG.md, and create a git tag. Then push the changes and the git tag with git push origin main --tags.

Metadata

Release files for wtpsplit-lite 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for wtpsplit-lite 0.2.0
File Size Uploaded
wtpsplit_lite-0.2.0.tar.gz 35.9 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for wtpsplit-lite 0.2.0
File Interpreter ABI Platform
wtpsplit_lite-0.2.0-py3-none-any.whl Python 3 none any Details

Total release size: 51.9 kB

Release files / wtpsplit_lite-0.2.0.tar.gz

Download URL wtpsplit_lite-0.2.0.tar.gz
Size 35.9 kB
Tags Source
SHA-256 checksum
How to use checksums
aaf9168cdbba9a20671df93dea0bd6ffb6f3a39f1b74cdd86ab3e81a073e97e8
BLAKE2b-256 checksum
How to use checksums
98792f74be2db6b03041fc3b256696da66c7dfd6ae2552907f950b7e071659f2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.7.13

Release files / wtpsplit_lite-0.2.0-py3-none-any.whl

Download URL wtpsplit_lite-0.2.0-py3-none-any.whl
Size 16.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3277a711f5c4c16162901fc37426557f4eb6eb6d90ba8fca276433fa9aa5fa4c
BLAKE2b-256 checksum
How to use checksums
3420f9453db785f3dbb4dd6b44aa073a3b2fcbde5e91b8746c769d348345d105
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.7.13

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files

0.1.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page