Skip to main content

SID SDK

sid-sdk turns search results into compact, model-facing document views. It assigns stable short IDs, selects relevant snippets, tracks the exact character ranges already shown, masks repeated text, and renders SID's <doc> format.

Snippet selection is implemented in Rust with Alyze Unicode UAX #29 tokenization and separate unigram and bigram BM25 scores. Returned ranges are Python character offsets into the original text.

Installation

pip install sid-sdk

Prebuilt ABI3 wheels require no Rust compiler on supported macOS, Linux, and Windows platforms. Building from the source distribution requires Python 3.12+ and Rust 1.88 or newer.

Basic use

from sid import DocumentCache

cache = DocumentCache(language="english")
cache.add_document(
    "database-id",
    {"title": "Example", "content": "The complete document text ..."},
)
view = cache.apply_snippet(
    "database-id",
    snippet_field="content",
    query="complete document",
)
print(view.render_xml())
cache.update_seen(view)

Named analyzers are available for Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. They lowercase tokens and remove Alyze's language-specific stopwords. Use language="generic" for lowercase Unicode tokenization without stopword removal. Chinese, Japanese, and Korean text is handled through generic UAX #29 segmentation; there are no dedicated language entries. apply_snippet accepts a one-call language= override, and forks inherit the cache default.

If every query token is a stopword, all windows have the same score and the selector deliberately returns the beginning of the document.

Model-facing IDs

New caches assign six ASCII digit strings, from "000000" through "999999", instead of the previous five lowercase letters. IDs remain strings: preserve leading zeros in storage, JSON tool schemas, and model responses. For example, "004271" and the ranged reference "004271#10:70" identify the same document. Database IDs are unchanged. Parsers still accept older letter-based references; resolving any reference requires its original cache mapping.

Each cache family can mint 1,000,000 distinct IDs over its lifetime, down from 11,881,376. Exhaustion raises IdSpaceExhausted without recycling IDs. Forks share the stream and mappings; independent caches can assign the same ID to different documents, so keep references scoped to their cache family. Check model-provided IDs with cache.contains_model_facing_id before using them.

This default prioritizes reference accuracy based on observed model behavior. Token cost depends on the tokenizer: the repository's historical measurements were about six tokens per numeric ID on Qwen and two on o200k/cl100k, versus about three for five-letter IDs. Downstream letter-only validators, prompt examples, and seeded snapshots need updating; the same seed now produces a different default sequence. Explicit custom IdStream(alphabet=..., length=...) settings continue to work, including the old lowercase alphabet with length 5.

Character ranges

DocumentCache(range_mode="lenient") is the default. A partially overlapping range is reduced to its exact intersection with the document, while an empty, inverted, malformed, or wholly disjoint range raises InvalidCharacterRange. Use range_mode="strict" to require 0 <= start < end <= len(content) exactly. Ranges are half-open Python character offsets and are never expanded to word boundaries, reversed, or replaced with the complete document.

Call cache.resolve_char_range(data_id, snippet_field, (start, end)) before passing a model-provided range onward. get_single_span_document_view applies the same configured policy, and rendered IDs always describe the exact slice displayed.

Rendering and forks

XML document bodies and attribute values escape &, <, and >; quotes are left unchanged. Falsy attributes are omitted from XML. Markdown tables stringify their values normally, so empty strings make empty cells while 0 and False remain visible. Snippet-less views require explicit display_fields.

Forks share one document store, ID mapping, and ID stream. A document added by any family member is immediately readable by the others, and adding the same data_id again is an idempotent lookup. Each fork keeps its own seen ledger, initialized from the parent's current ledger, so agents can track what they have shown independently.

Concurrent document additions are safe across a fork family, and separate forks may run on separate threads. A single fork is intended for one agent; apply_snippet and update_seen are separate ledger operations and are not an atomic read-modify-write pair when called concurrently on the same fork.

Development

uv sync --group dev
uv run maturin develop --locked
uv run pytest -rxX --strict-markers
cargo test --locked

License

MIT. See LICENSE and THIRD_PARTY_NOTICES.md.

Metadata

Release files for sid-sdk 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sid-sdk 0.2.1
File Size Uploaded Provenance Actions
sid_sdk-0.2.1.tar.gz 92.4 kB With provenance Details for sid_sdk-0.2.1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for sid-sdk 0.2.1
File Actions
sid_sdk-0.2.1-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for sid_sdk-0.2.1-cp312-abi3-win_amd64.whl
sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_x86_64.whl CPython 3.12 abi3 Linux musl 1.2+ x86-64 Details for sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_x86_64.whl
sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_aarch64.whl CPython 3.12 abi3 Linux musl 1.2+ ARM64 Details for sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_aarch64.whl
sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.12 abi3 Linux glibc 2.17+ x86-64 Details for sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.12 abi3 Linux glibc 2.17+ ARM64 Details for sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sid_sdk-0.2.1-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for sid_sdk-0.2.1-cp312-abi3-macosx_11_0_arm64.whl
sid_sdk-0.2.1-cp312-abi3-macosx_10_12_x86_64.whl CPython 3.12 abi3 macOS 10.12+ x86-64 Details for sid_sdk-0.2.1-cp312-abi3-macosx_10_12_x86_64.whl

Total release size: 2.9 MB

Release files / sid_sdk-0.2.1.tar.gz

Download URL sid_sdk-0.2.1.tar.gz
Size 92.4 kB
Tags Source
SHA-256 checksum
How to use checksums
c8c0a4c212556cb90c0773832621aa9d3eabd3d7e3f8392e31da19cf1aee00c3
BLAKE2b-256 checksum
How to use checksums
6d13255cfd4c2eecb7a272ef60e6742cb9d0ff065ba4c76df8bf9d547ab71cf2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-win_amd64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-win_amd64.whl
Size 260.2 kB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
e5a00cc38ae5021add0c359047256ea919e77c2655ef7724146b668daf074d16
BLAKE2b-256 checksum
How to use checksums
4112c19f5a95c600b27d80cc399fb504a5495875b09f0e2b0bf2c7a5701e52cb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_x86_64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_x86_64.whl
Size 591.4 kB
Tags CPython 3.12 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
56636624bb4f10f5701888206a7e89e55f8d124571b6bec08d00478a752a92af
BLAKE2b-256 checksum
How to use checksums
c0838ba7ad140f290341a3ca9becd209aba247216e34eaf23ac95a7cd2fd2844
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_aarch64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-musllinux_1_2_aarch64.whl
Size 543.6 kB
Tags CPython 3.12 Linux musl 1.2+ ARM64 abi3
SHA-256 checksum
How to use checksums
32a6a2650442e1dd3e07a7569c0537c91fdf6194c212f1b35923555db9523b45
BLAKE2b-256 checksum
How to use checksums
47bd38cf6ac15fde0d2fba4c9a7bc06f4d6003782bcf9316dceab3dc1e6f7125
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 380.6 kB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
ec09a3dfab5b6414250811049d90672dfe7b61821f346a7cd5492ad91fc928cd
BLAKE2b-256 checksum
How to use checksums
90a17f46ee64459adfaf3eaf4fe347c3d3a8eb624930fc3aa833dc5191f567ef
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 365.6 kB
Tags CPython 3.12 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
b8a204800acedb2bdf0d3f409be00a9944f050624e1ef20d7a51be9778168ebd
BLAKE2b-256 checksum
How to use checksums
e45b625c46ce433bbebd610a13578474081441e884812d2399258361c683a89a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-macosx_11_0_arm64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-macosx_11_0_arm64.whl
Size 326.6 kB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
bf2c6947f30ba71d869f3889df4c7d52561e3d3980df8c0517941e2f9d5543c4
BLAKE2b-256 checksum
How to use checksums
c3c5d966082e25f16eff8ed72b45587a5f2034389b27ddca5a14f47b74056c74
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Release files / sid_sdk-0.2.1-cp312-abi3-macosx_10_12_x86_64.whl

Download URL sid_sdk-0.2.1-cp312-abi3-macosx_10_12_x86_64.whl
Size 333.9 kB
Tags CPython 3.12 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
21d4ea3ec81e918019b9e1419b668c088bdba4f47a74b67693505dd0e0a824b8
BLAKE2b-256 checksum
How to use checksums
3d59bcdbe51a7cdeabc458cfbddba5e402fbde737b5f85ff01082c5ef4db61e2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

Security for sid-sdk 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 8 files in this release have been uploaded with provenance metadata.

Select from 8 files

Attestation for sid_sdk-0.2.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 8, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page