stringzilla 1.0.3

Crunch 100+ GB Strings in Python with ease

Stringzilla 🦖

Crunch 100+ GB Strings in Python with ease, leveraging SIMD Assembly

Stringzilla was born many years ago as a tutorial for SIMD accelerated string-processing. But one day, processing 100+ GB of Chemistry and AI datasets, I transformed it into a library. It's designed to replace open(...).readlines(), str().splitlines(), and many other typical workloads with very long strings.

Benchmark IoT Arm Laptop x86 Server Python: str.find 4 MB/s 14 MB/s 11 MB/s C++: std::string::find 560 MB/s 1,2 GB/s 1,3 GB/s Stringzilla 4,3 Gb/s 12 GB/s 12,1 GB/s

Usage

pip install stringzilla

There are two classes you can use interchangeably:

from stringzilla import Str, File, Strs

text: str = 'some-string'
text: Str = Str('some-string')
text: File = File('some-file.txt')

Once constructed, the following interfaces are supported:

len(text) -> int
'substring' in text -> bool
text[42] -> str

text.contains(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
) -> bool

text.find(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
) -> int

text.count(
    'subtring',
    start=0, # optional
    end=9223372036854775807, # optional
    **, # non-traditional arguments:
    allowoverlap=False, # optional
) -> int

text.splitlines(
    keeplinebreaks=False, # optional
    **, # non-traditional arguments:
    separator='\n', # optional
) -> Strs # similar to list[str]

text.split(
    separator=' ', # optional
    maxsplit=9223372036854775807, # optional
    **, # non-traditional arguments:
    keepseparator=False, # optional
) -> Strs # similar to list[str]

Development

rm -rf build && pip install -e . && pytest scripts/test.py -s -x

pip install -e . --no-index --no-deps

To benchmark on some custom file and pattern combinations:

python scripts/bench.py --haystack_path "your file" --needle "your pattern"

To benchmark on synthetic data:

python scripts/bench.py --haystack_pattern "abcd" --haystack_length 1e9 --needle "abce"

To validate packaging:

cibuildwheel --platform linux

Compiling C++ tests:

brew install libomp llvm
cmake -B ./build_release \
    -DCMAKE_C_COMPILER="/opt/homebrew/opt/llvm/bin/clang" \
    -DCMAKE_CXX_COMPILER="/opt/homebrew/opt/llvm/bin/clang++" \
    -DSTRINGZILLA_USE_OPENMP=1 \
    -DSTRINGZILLA_BUILD_TEST=1 \
    && \
    make -C ./build_release -j && ./build_release/stringzilla_test