stringzilla 0.1.1

Crunch 100+ GB Strings in Python with ease

Stringzilla

Crunch 100+ GB Strings in Python with ease, leveraging SIMD Assembly

Stringzilla was born many years ago as a tutorial for SIMD accelerated string-processing. But one day, processing 100+ GB Chemistry and AI datasets, I decided to transform it into a library. It's designed to replace open(...).readlines(), str().splitlines() and many other common workloads with very long strings.

Benchmark IoT Arm Laptop x86 Server Python: str.find 4 MB/s 14 MB/s 11 MB/s C++: std::string::find 560 MB/s 1,2 GB/s 1,3 GB/s Stringzilla 4,3 Gb/s 12 GB/s 12,1 GB/s

Usage

pip install stringzilla

There are two classes you can use interchangibly:

from stringzilla import Str, File, Slices

text: str = 'some-string'
text: Str = Str('some-string')
text: File = File('some-file.txt')

Once constructed, following interfaces are supported:

len(text) -> int
'substring' in text -> bool
text[42] -> str

text.contains(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
) -> bool

text.find(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
) -> int

text.count(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
    **, # non-traditional arguments:
    allowoverlap=False, # optional
) -> int

text.splitlines(
    keeplinebreaks=False, # optional
    **, # non-traditional arguments:
    separator='\n', # optional
) -> Slices # similar to list[str]

text.split(
    separator=' ', # optional
    maxsplit=9223372036854775807, # optional
    **, # non-traditional arguments:
    keepseparator=False, # optional
) -> Slices # similar to list[str]

Development

rm -rf build && pip install -e . && pytest scripts/test.py -s -x

To benchmark on some custom file and pattern combination:

python scripts/bench.py --path "your file" --pattern "your pattern"

To validate packaging:

cibuildwheel --platform linux