polars-text

Polars expression plugins for fast, practical text analysis. Use them as expressions or via the pl.col("text").text.* namespace, plus a few Series-based utilities for token frequency stats.

Quick start

import polars as pl
import polars_text

df = pl.DataFrame({
    "text": [
        "Alice said \"Hello world\".",
        "Hello again, world!",
    ]
})

out = df.with_columns([
    pl.col("text").text.clean_text().alias("clean"),
    pl.col("text").text.word_count().alias("word_count"),
    pl.col("text").text.char_count().alias("char_count"),
    pl.col("text").text.sentence_count().alias("sentence_count"),
    pl.col("text").text.tokenize(
        model="native:plain_words_en",
        lowercase=True,
        remove_punct=True,
    ).alias("tokens"),
])

Expressions and namespace

Tokenization is available through the text namespace on expressions.

Tokenization

• pl.col("text").text.tokenize(model="native:plain_words_en", lowercase=True, remove_punct=True, cache=None)
• pl.col("text").text.embedding(embedder_model=None, cache=None, batch_size=None)
• clean_text(expr)
• word_count(expr)
• char_count(expr)
• sentence_count(expr)
• concordance(expr, search_word, num_left_tokens=5, num_right_tokens=5, regex=False, case_sensitive=False, remove_punct=False)

Namespace usage

df = pl.DataFrame({"text": ["Hello world, hello again."]})

out = df.select([
    pl.col("text").text.clean_text().alias("clean"),
    pl.col("text").text.word_count().alias("word_count"),
    pl.col("text").text.tokenize(model="native:plain_words_en").alias("tokens"),
])

tokenize returns a list of structs with token, start, and end character offsets. Pass an explicit native:, huggingface:, or lindera: model ID. Pass cache=Path("tokens.duckdb") to persist tokenization results in a DuckDB cache and reuse them by content hash; leave cache=None to compute directly through the Rust plugin.

Pass remove_punct=True to concordance to exclude punctuation and symbol-only tokens from context counts and L1/R1. The returned contexts retain the original punctuation and whitespace between lexical tokens and the match; literal and regular-expression matching are unchanged.

Embeddings

embedding accepts a string expression or a list-of-string expression. String input returns List(Float32) per row; list input returns nested List(List(Float32)) per row.

df = pl.DataFrame({"text": ["A short document."], "chunks": [["first", "second"]]})

out = df.select([
    pl.col("text").text.embedding(cache="embeddings.duckdb").alias("embedding"),
    pl.col("chunks").text.embedding(cache="embeddings.duckdb").alias("chunk_embeddings"),
])

The Rust plugin downloads and loads Hugging Face ONNX sentence-transformer repositories automatically through hf-hub. Repositories without ONNX files are not supported. Passing cache=Path("embeddings.duckdb") persists vectors in a separate DuckDB cache keyed by model, revision, execution-provider label, and text hash.

Concordance

Get left/right context windows around a search term. Output is a list of structs that you can explode and unnest for tabular use.

df = pl.DataFrame({"text": ["Hello world, hello again."]})

concordance = (
    pl.col("text")
    .text.concordance("hello", num_left_tokens=1, num_right_tokens=1)
    .list.explode()
    .struct.unnest()
)

out = df.select(concordance)

Topic modelling

topic_modeling consumes a complete document column and returns one scalar run result. The result keeps document outcomes separate from complete topic metadata:

{
  documents: [{doc_index, dominant_topic, topic_distribution}],
  topics: [{id, representative_words, x, y}],
  n_chunks,
  truncated_segment_count,
  stage_timings_ms
}

Automatic, Paragraph, and Sentence modes differ only when constructing Topic Segments. All modes then share embedding, clustering, c-TF-IDF, and document rollup. Clustering treats every segment as one observation. Rollup weights each segment by the Unicode-character length of its retained text; Automatic overlap counts repeated text again.

Token frequencies and stats

Compute corpus token counts and compare corpora with standard statistics.

series_0 = pl.Series("text", ["hello world", "hello again"])
series_1 = pl.Series("text", ["goodbye world"])

freqs_0 = pt.token_frequencies(series_0, model="native:plain_words_en")
freqs_1 = pt.token_frequencies(series_1, model="native:plain_words_en")

stats = pt.token_frequency_stats(freqs_0, freqs_1)

Output schemas

Tokenization (list of structs):

• token
• start
• end

Concordance (list of structs):

• left_context, matched_text, right_context
• start_idx, end_idx
• l1, r1 (first token on left/right for quick filtering)

Models and downloads

Some features download tokenizer assets on first use and run on CPU:

• Hugging Face tokenizers: for example huggingface:bert-base-uncased (tokenizer.json via hf-hub)
• Lindera dictionaries: lindera:cc-cedict, lindera:jieba, lindera:ja-ipadic, lindera:ja-ipadic-neologd, lindera:ja-unidic, and lindera:ko-dic from official Lindera release zips

The initial call may take longer while models download and cache.

Embedding features download ONNX artifacts on first use. Some ONNX repositories store tensor data in sidecar files such as onnx/model.onnx_data; those files are fetched automatically when present. ONNX Runtime uses DirectML on Windows when available, XNNPACK acceleration on supported CPU platforms, and CPU fallback.

Development

Build the extension locally with maturin and then import as polars_text. See the repository-level development and release runbooks for complete procedures.

make build
make test

For faster Rust iteration, use feature-scoped targets such as make check-tokenization, make build-tokenization, or make build-topic. Leave JOBS unset for Cargo's default parallelism, or pass JOBS=<n> to cap it.