skeinrank 0.10.0

Dictionary-first extraction, alias canonicalization, and lightweight reranking utilities for technical documents.

skeinrank-core

skeinrank-core is the lightweight Python SDK and CLI for deterministic local terminology canonicalization.

It is the zero-friction entrypoint for SkeinRank: no Governance API, Elasticsearch, RabbitMQ, Celery, Docker, OpenRouter token, or ML dependencies are required.

30-second demo

import skeinrank

print(skeinrank.canonicalize("k8s pg timeout"))
# kubernetes postgresql timeout

print(skeinrank.extract("sev1 on kube after deploy"))
# ['critical incident', 'kubernetes', 'deployment']

The module-level helpers use a built-in platform_ops_demo dictionary so the first call works without a file. The demo dictionary is small enough to inspect, but expressive enough to show infrastructure, incidents, CI/CD, search, RAG, and context-shaped company language.

The same built-in dictionary also demonstrates why context matters:

import skeinrank

print(skeinrank.canonicalize("pg timeout"))
# postgresql timeout

print(skeinrank.canonicalize("pg layout"))
# page layout

print(skeinrank.canonicalize("pg dashboard"))
# product group

CLI from a source checkout:

poetry run skeinrank canonicalize "k8s pg timeout" --text
poetry run skeinrank extract "sev1 on kube after deploy" --text --compact

Install from a checkout

cd packages/skeinrank-core
poetry install
poetry run pytest -q

Legacy reranking modules remain in the source tree for compatibility, but the package no longer exposes heavyweight ML install extras. The local SDK facade, demo dictionary, CLI, and document helpers do not require ML dependencies.

Public Python facade

Use SkeinRank when you want to pass a dictionary in code:

from skeinrank import SkeinRank

sr = SkeinRank({
    "kubernetes": ["k8s", "kube", "kuber"],
    "postgresql": ["pg", "postgres", "psql"],
})

print(sr.canonicalize("kuber timeout on pg"))
# kubernetes timeout on postgresql

print(sr.extract("kuber timeout on pg"))
# ['kubernetes', 'postgresql']

Use explain=True when you need offsets, slots, and highlighted evidence:

result = sr.extract("k8s rollout uses pg", explain=True)

print(result.canonical_values)
print(result.matches[0].alias)
print(result.matches[0].highlighted_fragment)

The same facade can load a full SkeinRank dictionary JSON/YAML file:

from skeinrank import SkeinRank

sr = SkeinRank.from_file("company.dictionary.yaml")
print(sr.canonicalize("k8s rollout uses pg database"))

Built-in demo dictionary and examples

The built-in platform_ops_demo dictionary contains more than 30 canonical terms and more than 80 aliases across platform operations, incidents, CI/CD, search, RAG, and SkeinRank concepts. It is intentionally not a production vocabulary; it is a compact first-touch dictionary for demos, tests, tutorials, and screenshots.

Useful demo phrases:

Input	Output
k8s pg timeout	kubernetes postgresql timeout
sev1 on kube after pg migration	critical incident on kubernetes after postgresql database migration
gha deploy hit rmq latency spike	github actions, deployment, message queue, latency
pg layout	page layout
pg dashboard	product group

Examples live in ../../examples/sdk:

• zero_friction_demo.py runs the facade from Python.
• platform_ops_demo.dictionary.json exports the built-in dictionary in the public dictionary shape.

Dictionary-first SDK

The lower-level dictionary SDK remains available for callers that already use the governance export or skeinrank-migrate dictionary shape.

from skeinrank import load_dictionary, extract_terms, canonicalize_text

dictionary = load_dictionary("../../examples/migration/console_dictionary.example.json")

result = extract_terms(
    "This instruction helps deploy 500 k8s servers backed by Postgres.",
    dictionary=dictionary,
)

print(result.canonical_values)  # ['kubernetes', 'postgresql']

canonicalized = canonicalize_text(
    "k8s rollout uses pg database",
    dictionary=dictionary,
)
print(canonicalized.text)  # kubernetes rollout uses postgresql database

Stable dictionary exports include:

• SkeinRank, canonicalize(...), extract(...), demo_dictionary(...), demo_dictionary_payload(...)
• Dictionary, DictionaryTerm, DictionaryAlias, DictionaryStopListEntry
• load_dictionary(...), validate_dictionary(...)
• extract_terms(...), canonicalize_text(...)
• ExtractionResult, TermMatch, CanonicalizedText

The matcher is deterministic and local. It honors active/deprecated term and alias statuses, profile/global stop lists, returns offsets, and includes evidence snippets with <mark>...</mark> highlights.

Document text extraction utilities

Local document helpers can extract text before running the SDK matcher. They do not require the Governance API, Elasticsearch, Celery, or a database.

from skeinrank import load_document_text, extract_terms_from_document

text = load_document_text("incident-runbook.md")
result = extract_terms_from_document(
    "incident-runbook.md",
    dictionary="../../examples/migration/console_dictionary.example.json",
)

print(result.document.file_name)
print(result.extraction.canonical_values)

Supported formats without extra dependencies:

• text-like files: .txt, .md, .rst, .log, .csv, .tsv, .json, .jsonl, .yaml, .yml
• .html / .htm with scripts/styles ignored
• .docx via a small stdlib ZIP/XML reader

PDF extraction is supported when the caller installs pypdf in the environment. The core package does not require it by default.

Local CLI

Validate a dictionary exported from the governance API or used by skeinrank-migrate:

poetry run skeinrank validate-dictionary ../../examples/migration/console_dictionary.example.json
poetry run skeinrank validate-dictionary ../../examples/migration/console_dictionary.example.yaml --json

Run zero-config demo extraction/canonicalization:

poetry run skeinrank extract "k8s rollout uses pg database" --text --compact
poetry run skeinrank canonicalize "k8s rollout uses pg database" --text

Print or export the built-in demo dictionary:

poetry run skeinrank demo-dictionary --compact
poetry run skeinrank demo-dictionary --output ../../examples/sdk/platform_ops_demo.dictionary.json

Run the example script:

poetry run python ../../examples/sdk/zero_friction_demo.py

Run against a specific dictionary file:

poetry run skeinrank extract "k8s rollout uses pg database" \
  --text \
  --dictionary ../../examples/migration/console_dictionary.example.json

poetry run skeinrank canonicalize incident-runbook.md \
  --dictionary ../../examples/migration/console_dictionary.example.json \
  --output incident-runbook.canonicalized.txt

Extract plain text from a document before matching:

poetry run skeinrank document-text incident-runbook.docx --output incident-runbook.txt

The CLI returns JSON for extract, raw text by default for canonicalize and document-text, and supports --output, --compact, --max-matches, and --context-chars where relevant.

Attribute extraction and enrichment

The older attribute/profile API is still available for advanced local enrichment workflows.

from skeinrank import build_attribute_profile, enrich_texts

profile = build_attribute_profile(
    profile_id="company_terms",
    aliases={
        "kubernetes": ["k8s", "kube", "kuber"],
        "postgresql": ["pg", "postgres", "psql"],
    },
    slots={
        "kubernetes": "TOOL",
        "postgresql": "DB",
    },
    snapshot_version="company_terms@v1",
)

rows = enrich_texts(
    [
        {"id": "doc-1", "text": "k8s timeout after upgrade"},
        {"id": "doc-2", "text": "pg latency spike"},
    ],
    profile=profile,
)

print(rows[0]["canonical_values"])

Use this layer when you need profile templates, fuzzy alias fallback, richer passport/debug traces, or JSONL enrichment helpers.

Publishing checklist

The package is published through the manual publish-skeinrank-core GitHub Actions workflow. The recommended flow is:

1. Build and test locally.
2. Publish to TestPyPI.
3. Install from TestPyPI in a clean environment.
4. Publish to PyPI only after the TestPyPI smoke test passes.

Local packaging checks:

poetry install
poetry run pytest -q
poetry build
poetry run python -m pip install --upgrade twine
poetry run twine check dist/*

See docs/PUBLISHING.md for the full release checklist.

Public API policy

Only symbols re-exported from skeinrank.__init__ should be treated as stable public API. Internal modules may change without notice.