Skip to main content

A rule based sentence segmentation library.

Project description

cutters

A rule based sentence segmentation library.
Python bindings for the cutters library written in Rust.

Release License Downloads

🚧 This library is experimental. 🚧

Features

  • Full UTF-8 support.
  • Robust parsing.
  • Language specific rules (each defined by its own PEG).
  • Fast and memory efficient parsing via the pest library.
  • Sentences can contain quotes which can contain subsentences.

Supported languages

  • Croatian (standard)
  • English (standard)

There is also an additional Baseline "language" that simply splits the text on sentence terminals as defined by UTF-8. Its intended use is for benchmarking.

Example

After installing the cutters package with pip, usage is simple (note that the language is defined via ISO 639-1 two letter language codes).

import cutters

text = """
Petar Krešimir IV. je vladao od 1058. do 1074. St. Louis 9LX je događaj u svijetu šaha. To je prof.dr.sc. Ivan Horvat. Volim rock, punk, funk, pop itd. Tolstoj je napisao: "Sve sretne obitelji nalik su jedna na drugu. Svaka nesretna obitelj nesretna je na svoj način."
""";

sentences = cutters.cut(text, "hr");

print(sentences);

This results in the following output (note that the str struct fields are &str).

[Sentence {
    str: "Petar Krešimir IV. je vladao od 1058. do 1074. ",
    quotes: [],
}, Sentence {
    str: "St. Louis 9LX je događaj u svijetu šaha.",
    quotes: [],
}, Sentence {
    str: "To je prof.dr.sc. Ivan Horvat.",
    quotes: [],
}, Sentence {
    str: "Volim rock, punk, funk, pop itd.",
    quotes: [],
}, Sentence {
    str: "Tolstoj je napisao: \"Sve sretne obitelji nalik su jedna na drugu. Svaka nesretna obitelj nesretna je na svoj način.\"",
    quotes: [
        Quote {
            str: "Sve sretne obitelji nalik su jedna na drugu. Svaka nesretna obitelj nesretna je na svoj način.",
            sentences: [
                "Sve sretne obitelji nalik su jedna na drugu.",
                "Svaka nesretna obitelj nesretna je na svoj način.",
            ],
        },
    ],
}]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cutters-0.1.4.tar.gz (6.1 kB view hashes)

Uploaded Source

Built Distributions

cutters-0.1.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.8 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

cutters-0.1.4-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (274.0 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

cutters-0.1.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.6 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

cutters-0.1.4-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.9 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

cutters-0.1.4-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (294.7 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

cutters-0.1.4-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (274.8 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.4 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.6 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.3 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.7 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.3 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.7 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.7 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (274.0 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.5 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.9 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

cutters-0.1.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (292.5 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

cutters-0.1.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (273.9 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page