kaos-nlp-core

Part of Kelvin Agentic OS (KAOS) — open agentic infrastructure for legal work, built by 273 Ventures. See the full KAOS package map for the rest of the stack.

PyPI - Version Python License CI

kaos-nlp-core is a high-performance NLP primitives library for KAOS — a pure-Rust core with Python bindings via PyO3/Maturin. It provides the text-processing building blocks the rest of the stack relies on: SIMD-accelerated string operations, multi-pattern matching, finite-state transducers, sentence segmentation, BM25 retrieval, fuzzy hashing, and typed Python wrappers throughout.

It is dependency-light: the BASE install pulls kaos-nlp-core itself, numpy>=2.1 (used by similarity, chunker/aggregation marshalling, and retrieval helpers — see pyproject.toml:54), and the bundled Punkt sentence-segmenter model (~12 MB). Optional extras layer in the rest of the KAOS ecosystem.

Use and authorship disclosure

kaos-nlp-core provides deterministic primitives — segmentation, chunking, tokenization, search, aggregation, hashing — that take text in and produce text spans / scores / hashes back. No network calls, no LLM calls, no provider-side data transmission happens from this package; everything runs in-process. Downstream consumers (notably kaos-llm-core Programs) may transmit text derived from these primitives to LLM providers, so callers handling sensitive data should check the consuming package's data-handling disclosure.

This codebase is AI-assisted: substantial portions were generated with Claude (Anthropic) and human-reviewed before commit. Public behavior is covered by the test suite under tests/; large-corpus scale tests are opt-in (pytest tests/scale -m slow) and require the HF JSONL fixtures (USC, EDGAR, patents). Bug reports welcome via GitHub Issues; security reports follow SECURITY.md.

Install

uv add kaos-nlp-core
# or
pip install kaos-nlp-core

kaos-nlp-core requires Python 3.13 or newer. The published wheels are cp313-abi3 — one wheel per OS/architecture covers every CPython 3.13+ minor (3.13, 3.14, 3.15, …). No re-release needed when 3.15 ships.

Platform coverage: Linux x86_64 (manylinux), Linux aarch64 (manylinux), macOS arm64, Windows x86_64, Windows arm64. musllinux wheels were last published in 0.1.0a2 — Alpine users should pin <=0.1.0a2 for now.

Quick start

from kaos_nlp_core import tokenizer, algorithms

# Two output shapes for tokenization:
#   tokenize_words → list[str]        — just the surface forms (fastest)
#   tokenize       → list[TokenSpan]  — .text / .start / .end when you
#                                       need character offsets back into
#                                       the source string
words = tokenizer.tokenize_words("kaos-nlp-core ships fast NLP primitives.")
print(words)
# ['kaos-nlp-core', 'ships', 'fast', 'NLP', 'primitives']

for s in tokenizer.tokenize("kaos-nlp-core ships fast NLP primitives.")[:3]:
    print(f"{s.start}-{s.end}: {s.text!r}")
# 0-13: 'kaos-nlp-core'
# 14-19: 'ships'
# 20-24: 'fast'

# Multi-byte safe (CJK + emoji) — offsets are CHARACTER offsets, not bytes
for s in tokenizer.tokenize("東京 emoji 😀 test"):
    print(f"{s.start}-{s.end}: {s.text!r}")
# 0-2: '東京'
# 3-8: 'emoji'
# 9-10: '😀'
# 11-15: 'test'

# Algorithms always return rich typed results
result = algorithms.levenshtein("kitten", "sitting")
print(f"distance={result.distance} similarity={result.similarity:.4f}")
# distance=3.0 similarity=0.5714

# Readability: one-shot helpers for the common scores, or a full report
from kaos_nlp_core.readability import flesch_kincaid_grade, readability_report
print(round(flesch_kincaid_grade("The cat sat on the mat. The dog ate a bone."), 2))
# -1.65
report = readability_report("The cat sat on the mat. The dog ate a bone.")
print(report.counts.words, round(report.scores.flesch_reading_ease, 1))
# 11 116.7

The _words shortcut exists wherever skipping offsets is meaningful work (tokenization). Everywhere else — segmentation (segment_sentences, segment_paragraphs, segment_lines), pattern matching, similarity algorithms — the API only ships the rich typed shape, because the metadata is the value.

Concepts

The package is organized around a small set of typed primitives.

Concept What it is
Algorithms kaos_nlp_core.algorithms — Levenshtein, Hamming, Jaro-Winkler, longest common substring, edit-distance variants. SIMD fast paths via stringzilla; ASCII fast paths before Unicode fallbacks.
Tokenizer kaos_nlp_core.tokenizer — Unicode-aware word/sentence tokenization with byte→char offset translation via build_byte_to_char_table(). Multi-byte safe (Latin diacritics, CJK, emoji).
Segmentation kaos_nlp_core.segmentation — Punkt sentence segmenter (bundled model models/default.npkt.gz, ~12 MB Apache-2.0 NLTK port).
Matching kaos_nlp_core.matching — Aho-Corasick multi-pattern matching, FST-backed fuzzy lookup via Levenshtein automata, regex.
Search kaos_nlp_core.search — BM25 retrieval, Searcher, sentence/paragraph search; pickle-safe with KNC magic header for index files.
Structures kaos_nlp_core.structures — Vocabulary, InvertedIndex, SparseTermMatrix, SimilarityMatrix. Compact, pickle-safe, bincode-2.0 backed.
Hashing kaos_nlp_core.hashing — CTPH (context-triggered piecewise hashing) via blake3, MinHash, LSH index, near-duplicate grouping.
Lexicon kaos_nlp_core.lexicon — query expansion, semantic graph traversal, gazetteer lookups.
Documents kaos_nlp_core.documents — Document, DocumentCollection with JSONL / HuggingFace loaders.
Quality kaos_nlp_core.quality — text-quality heuristics (token ratios, Unicode block distribution).
Readability kaos_nlp_core.readability — Flesch, Flesch-Kincaid, ARI, Coleman-Liau, SMOG, Gunning Fog, Dale-Chall, LIX/RIX with verified formula provenance; Rust-backed counting, CMUdict-exact syllables with tuned heuristic fallback.

CLI

kaos-nlp-core ships a kaos-nlp administrative CLI plus an optional kaos-nlp-serve MCP server (loopback-only by default; --http requires KAOS_NLP_HTTP_TOKEN as an operator acknowledgement that a reverse proxy is fronting authentication):

kaos-nlp tokenize doc.txt --lowercase --json          # word tokenization with spans
kaos-nlp segment doc.txt --mode sentences             # sentence segmentation (Punkt)
kaos-nlp compare "Robert" "Rupert" --algorithm jaro-winkler
kaos-nlp find "pattern" doc.txt --case-insensitive    # SIMD substring search
kaos-nlp index build corpus.txt --output idx.kncidx   # native persisted index
kaos-nlp search --index idx.kncidx "query terms"      # ranked search (BM25 default)
kaos-nlp hash doc.txt --algorithm ctph                # fuzzy hash
kaos-nlp duplicates ./corpus/ --threshold 0.5         # near-duplicate detection
kaos-nlp encode "Robert" --algorithm soundex          # phonetic encoding
kaos-nlp vocab build doc.txt --type frequency         # build vocabulary
kaos-nlp analyze doc.txt --json                       # text statistics report
kaos-nlp readability doc.txt --json                   # Flesch/FK/SMOG/Fog scores

kaos-nlp-serve            # MCP server, stdio transport
kaos-nlp-serve --http     # MCP server, streamable HTTP (operator-token gated)

Every command supports --json for machine-readable output. CLI search reads both the native persisted index format (KNC) and legacy .json bundles.

Note: 17 MCP tools are registered by register_nlp_tools(). Until 0.1.0a2, the [mcp] extra is reserved but unpopulated — manually run pip install kaos-core kaos-mcp before using kaos-nlp-serve. Once siblings publish to PyPI, pip install kaos-nlp-core[mcp] will cover the full install. Until then kaos-nlp-serve exits with an actionable install hint if kaos-core or kaos-mcp are missing.

Compatibility & status

Aspect
Python 3.13, 3.14 (informational matrix entries for 3.14t free-threaded and 3.15-dev). One cp313-abi3 wheel per OS/arch covers all 3.13+ minors.
OS Linux (manylinux, x86_64 + aarch64), macOS arm64, Windows x86_64, Windows arm64. macOS x86_64 deliberately skipped (Apple ended Intel sales in 2023); musllinux last shipped in 0.1.0a2.
Maturity Alpha. The public API is documented in kaos_nlp_core.__all__.
Stability policy Pre-1.0: minor bumps may change behaviour. Every change is documented in CHANGELOG.md.
Test coverage 298 Rust unit tests + Python pytest suite. Round-trip offset tests cover ASCII, multi-byte Latin, CJK, and emoji.
Type checker Validated with ty, Astral's Python type checker.

Companion packages

kaos-nlp-core is one of the packages in the Kelvin Agentic OS. The broader stack:

Package Layer What it does
kaos-core Core Foundational runtime, MCP-native types, registries, execution engine, VFS
kaos-content Core Typed document AST: Block/Inline, provenance, views
kaos-mcp Bridge FastMCP server, kaos management CLI, MCP resource templates
kaos-pdf Extraction PDF → AST with provenance
kaos-web Extraction Web extraction, browser automation, search, domain intelligence
kaos-office Extraction DOCX / PPTX / XLSX readers + writers to AST
kaos-tabular Extraction DuckDB-powered SQL analytics
kaos-source Data Government + financial data connectors (Federal Register, eCFR, EDGAR, GovInfo, PACER, GLEIF)
kaos-llm-client LLM Multi-provider LLM transport
kaos-llm-core LLM Typed LLM programming (Signatures, Programs, Optimizers)
kaos-nlp-core Primitives (Rust) High-performance NLP primitives
kaos-nlp-transformers ML Dense embeddings + retrieval
kaos-graph Primitives (Rust) Graph algorithms + RDF/SPARQL
kaos-ml-core Primitives (Rust) Classical ML on the document AST
kaos-citations Legal Legal citation extraction, resolution, verification
kaos-agents Agentic Agent runtime, memory, recipes
kaos-reference Sample Reference module for module authors

Packages depend on kaos-core; everything else is opt-in. Mix and match the ones you need.

Development

git clone https://github.com/273v/kaos-nlp-core
cd kaos-nlp-core
uv sync --group dev
uv run maturin develop --release

Install pre-commit hooks (recommended — they run the same checks as CI on every commit, scoped to staged files):

uvx pre-commit install
uvx pre-commit run --all-files     # one-time full sweep

Manual QA commands (the same set CI runs):

cargo fmt --check
cargo clippy --no-default-features --all-targets -- -D warnings
cargo test --no-default-features --lib
uv run ruff format --check python/kaos_nlp_core tests
uv run ruff check python/kaos_nlp_core tests
uv run ty check python/kaos_nlp_core tests
uv run pytest tests/

Build from source

uv build
uv pip install dist/*.whl

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for setup, quality gates, pull request expectations, and engineering standards. By contributing you agree to follow the project conduct expectations and certify the Developer Certificate of Origin v1.1 — sign every commit with git commit -s. Please open an issue before starting on a non-trivial change so we can align on scope.

Security

For security issues, please do not file a public issue. Report privately via GitHub Private Vulnerability Reporting or email security@273ventures.com. See SECURITY.md for the full disclosure policy.

License

Apache License 2.0 — see LICENSE and NOTICE.

Copyright 2026 273 Ventures LLC. Built for kelvin.legal.

Project links

Data verified by PyPI on Jul 22, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.10)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mjbommar273 from gravatar.com mjbommar273

Credits

Author: 273 Ventures LLC

GitHub Statistics

Data verified by PyPI on Jul 22, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.13

Tags

nlp tokenizer bm25 fst rust pyo3

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kaos_nlp_core-0.1.7.tar.gz (59.5 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

kaos_nlp_core-0.1.7-cp313-abi3-win_arm64.whl (49.3 MB view details)

Uploaded CPython 3.13+Windows ARM64

kaos_nlp_core-0.1.7-cp313-abi3-win_amd64.whl (49.5 MB view details)

Uploaded CPython 3.13+Windows x86-64

kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_x86_64.whl (50.2 MB view details)

Uploaded CPython 3.13+manylinux: glibc 2.28+ x86-64

kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_aarch64.whl (49.6 MB view details)

Uploaded CPython 3.13+manylinux: glibc 2.28+ ARM64

kaos_nlp_core-0.1.7-cp313-abi3-macosx_11_0_arm64.whl (49.7 MB view details)

Uploaded CPython 3.13+macOS 11.0+ ARM64

File details

Details for the file kaos_nlp_core-0.1.7.tar.gz.

File metadata

  • Download URL: kaos_nlp_core-0.1.7.tar.gz
  • Upload date:
  • Size: 59.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for kaos_nlp_core-0.1.7.tar.gz
Algorithm Hash digest
SHA256 08cd07f9b0536642043be55d2a30dfec515064c58ac81ad1e57e5a70d6a86788
MD5 6c07491648f05f69a7c55cfbca32444a
BLAKE2b-256 c9b3454e131c85c0f3ec92999f1c42127c7935190f3e09135dbf22b027ff265c

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7.tar.gz:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file kaos_nlp_core-0.1.7-cp313-abi3-win_arm64.whl.

File metadata

File hashes

Hashes for kaos_nlp_core-0.1.7-cp313-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 566de7a1dd80e7398f22cfff4d7d17708fb343d1581ad1812bff29221c168304
MD5 564fafa08d1ea8d2db6e8b9a0c59f3b8
BLAKE2b-256 2e4881017e38a7624987e580e98ca7986c13b49b415afa57d32d6166ec65ad8a

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7-cp313-abi3-win_arm64.whl:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file kaos_nlp_core-0.1.7-cp313-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for kaos_nlp_core-0.1.7-cp313-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 4b924943a3aa22170d24b0c92fb32b594b013ac26230c2eebcdd3d46bac75046
MD5 f57307fecbd012a364935f2f72a3763e
BLAKE2b-256 154a8447e8fae48864cc4d0a1ce367c40948e67e0d1490c31d546c3ce6d65823

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7-cp313-abi3-win_amd64.whl:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 ecbe48568035e38f876fb9d8a53cb07dd111e45f7931c0feca5524cfd3a8ede9
MD5 e5a7e8e8ffbf97398181a5f041b84430
BLAKE2b-256 7f6ca0a5492bc655e77ff21788e25136068160652ffc7f29b33d2d5f22892d5f

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_x86_64.whl:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 263f00c5de466d6f18e435bab41b8583065a83a1c0c296f3d99313804f89200f
MD5 676a000906e36cffeb7ed94cafc39517
BLAKE2b-256 e73f4cad341d357bef5149b1218d1aaa85e2a4ea69414cf8cab3539dd941a8ff

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7-cp313-abi3-manylinux_2_28_aarch64.whl:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file kaos_nlp_core-0.1.7-cp313-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for kaos_nlp_core-0.1.7-cp313-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 64fa6330eb379c021812652aa5cbc7b2a8fcb6154b235e424977448d66a4b332
MD5 e35b527c799eda080d50ffab5da4d32c
BLAKE2b-256 bcf1c1d9e4b2ec8d5819ef23e86aa5775e802566600f4c8cd2716164f226852b

See more details on using hashes here.

Provenance

The following attestation bundles were made for kaos_nlp_core-0.1.7-cp313-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on 273v/kaos-nlp-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page