polars-text 0.4.0

Polars expression plugins for text analysis

polars-text

Polars expression plugins for fast, practical text analysis. Use them as expressions or via the pl.col("text").text.* namespace, plus a few Series-based utilities for token frequency stats.

Quick start

import polars as pl
import polars_text

df = pl.DataFrame({
    "text": [
        "Alice said \"Hello world\".",
        "Hello again, world!",
    ]
})

out = df.with_columns([
    pl.col("text").text.clean_text().alias("clean"),
    pl.col("text").text.word_count().alias("word_count"),
    pl.col("text").text.char_count().alias("char_count"),
    pl.col("text").text.sentence_count().alias("sentence_count"),
    pl.col("text").text.tokenize(
        model="native:plain_words_en",
        lowercase=True,
        remove_punct=True,
    ).alias("tokens"),
])

Expressions and namespace

Tokenization is available through the text namespace on expressions.

Tokenization

• pl.col("text").text.tokenize(model="native:plain_words_en", lowercase=True, remove_punct=True, cache=None)
• pl.col("text").text.embedding(embedder_model=None, cache=None, batch_size=None)
• clean_text(expr)
• word_count(expr)
• char_count(expr)
• sentence_count(expr)
• concordance(expr, search_word, num_left_tokens=5, num_right_tokens=5, regex=False, case_sensitive=False)

Namespace usage

df = pl.DataFrame({"text": ["Hello world, hello again."]})

out = df.select([
    pl.col("text").text.clean_text().alias("clean"),
    pl.col("text").text.word_count().alias("word_count"),
    pl.col("text").text.tokenize(model="native:plain_words_en").alias("tokens"),
])

tokenize returns a list of structs with token, start, and end character offsets. Pass an explicit native:, huggingface:, or lindera: model ID. Pass cache=Path("tokens.duckdb") to persist tokenization results in a DuckDB cache and reuse them by content hash; leave cache=None to compute directly through the Rust plugin.

Embeddings

embedding accepts a string expression or a list-of-string expression. String input returns List(Float32) per row; list input returns nested List(List(Float32)) per row.

df = pl.DataFrame({"text": ["A short document."], "chunks": [["first", "second"]]})

out = df.select([
    pl.col("text").text.embedding(cache="embeddings.duckdb").alias("embedding"),
    pl.col("chunks").text.embedding(cache="embeddings.duckdb").alias("chunk_embeddings"),
])

The Rust plugin downloads and loads Hugging Face ONNX sentence-transformer repositories automatically through hf-hub. Repositories without ONNX files are not supported. Passing cache=Path("embeddings.duckdb") persists vectors in a separate DuckDB cache keyed by model, revision, execution-provider label, and text hash.

Concordance

Get left/right context windows around a search term. Output is a list of structs that you can explode and unnest for tabular use.

df = pl.DataFrame({"text": ["Hello world, hello again."]})

concordance = (
    pl.col("text")
    .text.concordance("hello", num_left_tokens=1, num_right_tokens=1)
    .list.explode()
    .struct.unnest()
)

out = df.select(concordance)

Token frequencies and stats

Compute corpus token counts and compare corpora with standard statistics.

series_0 = pl.Series("text", ["hello world", "hello again"])
series_1 = pl.Series("text", ["goodbye world"])

freqs_0 = pt.token_frequencies(series_0, model="native:plain_words_en")
freqs_1 = pt.token_frequencies(series_1, model="native:plain_words_en")

stats = pt.token_frequency_stats(freqs_0, freqs_1)

Output schemas

Tokenization (list of structs):

• token
• start
• end

Concordance (list of structs):

• left_context, matched_text, right_context
• start_idx, end_idx
• l1, r1 (first token on left/right for quick filtering)

Models and downloads

Some features download tokenizer assets on first use and run on CPU:

• Hugging Face tokenizers: for example huggingface:bert-base-uncased (tokenizer.json via hf-hub)
• Lindera dictionaries: lindera:cc-cedict, lindera:jieba, lindera:ja-ipadic, lindera:ja-ipadic-neologd, lindera:ja-unidic, and lindera:ko-dic from official Lindera release zips

The initial call may take longer while models download and cache.

Embedding features download ONNX artifacts on first use. Some ONNX repositories store tensor data in sidecar files such as onnx/model.onnx_data; those files are fetched automatically when present. ONNX Runtime uses DirectML on Windows when available, XNNPACK acceleration on supported CPU platforms, and CPU fallback.

Development

Build the extension locally with maturin and then import as polars_text.

For release and publishing procedures, see PUBLISH.md.

make build
make test

For faster Rust iteration, use feature-scoped targets such as make check-tokenization, make build-tokenization, or make build-topic. Leave JOBS unset for Cargo's default parallelism, or pass JOBS=<n> to cap it.