kaos-nlp-core

Part of Kelvin Agentic OS (KAOS) — open agentic infrastructure for legal work, built by 273 Ventures. See the full KAOS package map for the rest of the stack.

kaos-nlp-core is a high-performance NLP primitives library for KAOS — a pure-Rust core with Python bindings via PyO3/Maturin. It provides the text-processing building blocks the rest of the stack relies on: SIMD-accelerated string operations, multi-pattern matching, finite-state transducers, sentence segmentation, BM25 retrieval, fuzzy hashing, and typed Python wrappers throughout.

It is dependency-light: the BASE install pulls kaos-nlp-core itself, numpy>=2.1 (used by similarity, chunker/aggregation marshalling, and retrieval helpers — see pyproject.toml:54), and the bundled Punkt sentence-segmenter model (~12 MB). Optional extras layer in the rest of the KAOS ecosystem.

Use and authorship disclosure

kaos-nlp-core provides deterministic primitives — segmentation, chunking, tokenization, search, aggregation, hashing — that take text in and produce text spans / scores / hashes back. No network calls, no LLM calls, no provider-side data transmission happens from this package; everything runs in-process. Downstream consumers (notably kaos-llm-core Programs) may transmit text derived from these primitives to LLM providers, so callers handling sensitive data should check the consuming package's data-handling disclosure.

This codebase is AI-assisted: substantial portions were generated with Claude (Anthropic) and human-reviewed before commit. Public behavior is covered by the test suite under tests/; large-corpus scale tests are opt-in (pytest tests/scale -m slow) and require the HF JSONL fixtures (USC, EDGAR, patents). Bug reports welcome via GitHub Issues; security reports follow SECURITY.md.

Install

uv add kaos-nlp-core
# or
pip install kaos-nlp-core

kaos-nlp-core requires Python 3.13 or newer. The published wheels are cp313-abi3 — one wheel per OS/architecture covers every CPython 3.13+ minor (3.13, 3.14, 3.15, …). No re-release needed when 3.15 ships.

Platform coverage: Linux x86_64 (manylinux), Linux aarch64 (manylinux), macOS arm64, Windows x86_64, Windows arm64. musllinux wheels were last published in 0.1.0a2 — Alpine users should pin <=0.1.0a2 for now.

Quick start

from kaos_nlp_core import tokenizer, algorithms

# Two output shapes for tokenization:
#   tokenize_words → list[str]        — just the surface forms (fastest)
#   tokenize       → list[TokenSpan]  — .text / .start / .end when you
#                                       need character offsets back into
#                                       the source string
words = tokenizer.tokenize_words("kaos-nlp-core ships fast NLP primitives.")
print(words)
# ['kaos-nlp-core', 'ships', 'fast', 'NLP', 'primitives']

for s in tokenizer.tokenize("kaos-nlp-core ships fast NLP primitives.")[:3]:
    print(f"{s.start}-{s.end}: {s.text!r}")
# 0-13: 'kaos-nlp-core'
# 14-19: 'ships'
# 20-24: 'fast'

# Multi-byte safe (CJK + emoji) — offsets are CHARACTER offsets, not bytes
for s in tokenizer.tokenize("東京 emoji 😀 test"):
    print(f"{s.start}-{s.end}: {s.text!r}")
# 0-2: '東京'
# 3-8: 'emoji'
# 9-10: '😀'
# 11-15: 'test'

# Algorithms always return rich typed results
result = algorithms.levenshtein("kitten", "sitting")
print(f"distance={result.distance} similarity={result.similarity:.4f}")
# distance=3.0 similarity=0.5714

# Readability: one-shot helpers for the common scores, or a full report
from kaos_nlp_core.readability import flesch_kincaid_grade, readability_report
print(round(flesch_kincaid_grade("The cat sat on the mat. The dog ate a bone."), 2))
# -1.65
report = readability_report("The cat sat on the mat. The dog ate a bone.")
print(report.counts.words, round(report.scores.flesch_reading_ease, 1))
# 11 116.7

No install needed to try it — uv run pulls the wheel on the fly:

uv run --with kaos-nlp-core python -c "
from kaos_nlp_core.readability import flesch_kincaid_grade, readability_report

text = 'Hello, world. Readability scoring is now built into kaos-nlp-core.'
print('Flesch-Kincaid grade:', round(flesch_kincaid_grade(text), 2))

for name, value in readability_report(text).scores.to_dict().items():
    print(f'{name:28s} {value if isinstance(value, bool) else round(value, 2)}')
"
# Flesch-Kincaid grade: 9.77
# flesch_reading_ease          33.07
# flesch_kincaid_grade         9.77
# automated_readability_index  8.56
# coleman_liau_index           12.25
# smog_index                   8.84
# gunning_fog                  6.24
# lix                          37.83
# rix                          1.5
# smog_valid                   False

(smog_valid: False is the honesty flag: SMOG's calibration assumes ≥30 sentences. Dale-Chall is omitted entirely until you supply a familiar-word list.)

The _words shortcut exists wherever skipping offsets is meaningful work (tokenization). Everywhere else — segmentation (segment_sentences, segment_paragraphs, segment_lines), pattern matching, similarity algorithms — the API only ships the rich typed shape, because the metadata is the value.

Concepts

The package is organized around a small set of typed primitives.

CLI

kaos-nlp-core ships a kaos-nlp administrative CLI plus an optional kaos-nlp-serve MCP server (loopback-only by default; --http requires KAOS_NLP_HTTP_TOKEN as an operator acknowledgement that a reverse proxy is fronting authentication):

kaos-nlp tokenize doc.txt --lowercase --json          # word tokenization with spans
kaos-nlp segment doc.txt --mode sentences             # sentence segmentation (Punkt)
kaos-nlp compare "Robert" "Rupert" --algorithm jaro-winkler
kaos-nlp find "pattern" doc.txt --case-insensitive    # SIMD substring search
kaos-nlp index build corpus.txt --output idx.kncidx   # native persisted index
kaos-nlp search --index idx.kncidx "query terms"      # ranked search (BM25 default)
kaos-nlp hash doc.txt --algorithm ctph                # fuzzy hash
kaos-nlp duplicates ./corpus/ --threshold 0.5         # near-duplicate detection
kaos-nlp encode "Robert" --algorithm soundex          # phonetic encoding
kaos-nlp vocab build doc.txt --type frequency         # build vocabulary
kaos-nlp analyze doc.txt --json                       # text statistics report
kaos-nlp readability doc.txt --json                   # Flesch/FK/SMOG/Fog scores

kaos-nlp-serve            # MCP server, stdio transport
kaos-nlp-serve --http     # MCP server, streamable HTTP (operator-token gated)

Every command supports --json for machine-readable output. CLI search reads both the native persisted index format (KNC) and legacy .json bundles.

The CLI also runs without installing, via uvx:

uvx --from kaos-nlp-core kaos-nlp readability doc.txt --json

Note: 17 MCP tools are registered by register_nlp_tools(). Until 0.1.0a2, the [mcp] extra is reserved but unpopulated — manually run pip install kaos-core kaos-mcp before using kaos-nlp-serve. Once siblings publish to PyPI, pip install kaos-nlp-core[mcp] will cover the full install. Until then kaos-nlp-serve exits with an actionable install hint if kaos-core or kaos-mcp are missing.

Compatibility & status

Companion packages

kaos-nlp-core is one of the packages in the Kelvin Agentic OS. The broader stack:

Packages depend on kaos-core; everything else is opt-in. Mix and match the ones you need.

Development

git clone https://github.com/273v/kaos-nlp-core
cd kaos-nlp-core
uv sync --group dev
uv run maturin develop --release

Install pre-commit hooks (recommended — they run the same checks as CI on every commit, scoped to staged files):

uvx pre-commit install
uvx pre-commit run --all-files     # one-time full sweep

Manual QA commands (the same set CI runs):

cargo fmt --check
cargo clippy --no-default-features --all-targets -- -D warnings
cargo test --no-default-features --lib
uv run ruff format --check python/kaos_nlp_core tests
uv run ruff check python/kaos_nlp_core tests
uv run ty check python/kaos_nlp_core tests
uv run pytest tests/

Build from source

uv build
uv pip install dist/*.whl

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for setup, quality gates, pull request expectations, and engineering standards. By contributing you agree to follow the project conduct expectations and certify the Developer Certificate of Origin v1.1 — sign every commit with git commit -s. Please open an issue before starting on a non-trivial change so we can align on scope.

Security

For security issues, please do not file a public issue. Report privately via GitHub Private Vulnerability Reporting or email security@273ventures.com. See SECURITY.md for the full disclosure policy.

License

Apache License 2.0 — see LICENSE and NOTICE.

Copyright 2026 273 Ventures LLC. Built for kelvin.legal.