sse_lib

Thin Python client for the Semantic Search Engine (SSE) REST API (src/sse_api/ in this repository).

sse_lib is a client-side layer only: it maps Python calls onto the HTTP endpoints of a running SSE backend and turns the JSON answers into small dataclasses. It does not import Django, torch, pymilvus or any other server dependency — the single runtime requirement is requests.

from sse_lib import SSEClient, SearchOptions, GenerativeOptions

client = SSEClient("http://localhost:8271", token="...")

client.create_collection(
    "my_docs",
    "My Documents",
    "Test collection",
    embedder="radlab/polish-bi-encoder-mean",
    reranker="radlab/polish-cross-encoder",
    index_type="HNSW",
    chunking={"max_tokens_in_chunk": 128, "number_of_overlap_tokens": 0},
)
client.upload_files("my_docs", ["invoice.pdf", "regulations.zip"])
client.index_documents("my_docs", ["Some text to index."], category="Notes")

found = client.search("my_docs", "policy on data retention",
                      SearchOptions(max_results=20, rerank_results=True))
answer = client.generate_answer(
    found.query_response_id,
    GenerativeOptions(
        generative_model="radlab/pLLama-3-8B-DPO-L",
        percentage_rank_mass=40,
    ),
)
print(answer.answer)

Chunking belongs to the collection, not to a single call: declare it with create_collection(..., chunking={...}) (or strict_chunking=True to forbid any deviation) and the indexing calls above may leave both keys out — the server takes the settings of the collection. client.embedders_info() returns max_model_length of every embedder, the largest chunk its model can take in (client.rerankers_info() the pair budget of the rerankers). Passing max_tokens_in_chunk / number_of_overlap_tokens per call, or once per client with SSEClient(..., default_indexing_options={...}), still works; a split which cannot exist (overlap >= chunk) raises SSEValueError before the request goes out. percentage_rank_mass of generate_answer is required by the backend and has no library-wide default on purpose.

Installation

From a checkout:

pip install ./sse_lib            # or: pip install -e ./sse_lib  (development)

Once published to a registry, pip install sse-lib is all that is needed — the package is standalone and versioned independently (sse_lib.__version__).

Python 3.9+, requests as the only runtime dependency.

Authentication

Mode	How
Pre-existing token (default DRF token auth)	SSEClient(host, token="...") — sent as Authorization: Token <value>
Login with user and password	SSEClient(host, username="user", password="...") — POST login runs lazily, before the first request
Keycloak / OAuth (backend started with ENV_USE_KC_AUTH=1)	client.login_url() → redirect the user → client.login_with_code(code, state); later client.refresh_access_token() and client.logout()
Custom scheme	SSEClient(host, token="...", token_type="Bearer")

Configuration

Constructor argument	Environment variable	Default	Meaning
api_host	SSE_API_HOST	–	http://localhost:8271; the scheme may be omitted (localhost → http, anything else → https)
api_prefix	SSE_API_PREFIX	api	Path between the host and the endpoints; set api/v1 for versioned deployments (main.api in django-config.json)
token	SSE_API_TOKEN	–	Value of the Authorization header
language	SSE_API_LANGUAGE	pl	Sent as lang on every call (pl / en)
timeout	SSE_API_TIMEOUT	60.0	Per-request timeout in seconds
verify, session, headers, max_retries	–	True, new session, {}, 2	TLS verification, shared requests.Session, extra headers, GET retries
default_search_options, default_indexing_options	–	–	Client-wide option defaults merged into every search / indexing call

SSEClient is a context manager; leaving the block closes the HTTP session.

Per-file upload metadata

Upload one whole document per file; SSE performs chunking using the same values as create_collection(..., chunking=...):

name = "doc:20261022:abc123.txt"
result = client.upload_files(
    "my_docs",
    [(name, "Whole document text".encode("utf-8"), "text/plain")],
    indexing_options={"max_tokens_in_chunk": 400, "number_of_overlap_tokens": 60},
    file_metadata={name: {
        "category": "doc",
        "options": {"source_file": "docs/guide.md", "scope": "project",
                    "doc_type": "markdown"},
    }},
)

file_metadata is a separate multipart JSON string, not an indexing option. When supplied, it must cover every non-ZIP file exactly once. Each entry requires category (string, including an empty string) and options (JSON object); other entry fields are rejected. Exact names are preferred; unique basename aliases are accepted. Missing/unknown/ambiguous names, duplicate aliases or JSON keys, invalid shapes and non-finite numbers are rejected. ZIP archives may be included in the batch, but neither archives nor their members may have metadata entries.

category is stored in Document.category; options is merged into Document.metadata_json, with supplied values winning over reader values. It is not stored under an extra options wrapper. Nested JSON values are preserved. Omit file_metadata (or pass None) for legacy behavior, including ZIP directory categories. The constructor and upload response are unchanged.

Both the API and Celery workers must run the updated server code. No schema migration or new dependency is required: metadata is persisted in the existing IndexingJob.request_payload before dispatch. Update/reinstall sse_lib in client environments as well; an older server does not support this contract.

For a no-server transport test, see tests/test_client.py::IndexingTest.test_worker_upload_bytes_with_file_metadata and tests/support.py: build_client constructs the real SSEClient(host, token=..., transport=Transport(host, session=FakeSession(...))). The fake session returns real requests.Response objects with the envelope {"status": true, "body": {"dir_hash": "h", "is_indexed": true}}.

API coverage

Client method	REST endpoint
create_collection(...)	POST new_collection
list_collections(), get_collection(name), collection_exists(name)	GET collections
list_categories(collection)	GET categories
list_documents(collection, with_disabled=False), list_document_details(collection)	GET documents
upload_files(collection, files, ...)	POST upload_and_index_files
document_content(collection, document=None, from_page=None, max_pages=None, chunk_type=None, text_id=None)	GET document_content (text_id alone is enough and returns the chunk's page with .highlight)
index_documents(collection, documents, ...)	POST add_and_index_texts
indexing_status(collection=None, job_id=None)	GET indexing_status
delete_texts(collection, chunk_ids)	POST delete_texts
delete_documents(collection, documents=None, document_ids=None)	POST delete_documents
delete_collection(collection)	POST delete_collection
set_documents_use_in_search(collection, documents=None, document_ids=None, use_in_search=False), deactivate_documents(...), activate_documents(...)	POST set_documents_use_in_search
list_question_templates()	GET question_templates
list_filter_options()	GET filter_options
search(collection, query, options, ...)	POST search_with_options
generate_answer(query_response_id, options, ...)	POST generative_answer
generative_answer_with_citations(query_response_id, options, ...)	POST generative_answer with numbered sources enabled
rewrite_query(query, generative_model=..., historical_messages=None)	POST rewrite_query
rate_answer(answer_response_id, value, value_max, comment=None)	POST rate_generative_answer
list_search_history(collection=None, query_response_id=None), get_search_history(query_response_id), delete_search_history(query_ids=None, query_response_ids=None)	GET search_history, GET get_search_history, POST delete_search_history
list_generative_models(), list_embedders(), list_rerankers()	GET generative_models, GET embedders, GET rerankers
embedders_info(), rerankers_info()	GET embedders, GET rerankers — models_info with vector_size, device and max_model_length
new_chat(...), send_chat_message(...), save_chat(...), get_chat_by_hash(...), list_chats()	POST new_chat, POST add_user_message, POST save_chat, GET get_chat_by_hash, GET chats
login(), login_with_code(), refresh_access_token(), logout(), login_url()	POST login, POST refresh_token, POST logout, POST login_url
health(), is_healthy()	GET healthz (public — no token, no login)
request(method, endpoint, ...), get(...), post(...)	any endpoint, including ones this release does not wrap

Collections may be passed as a name, a Collection returned by create_collection() / list_collections(), or any mapping with a name key. A Collection also carries the settings of the collection: metadata_json, chunking ({} when none was declared) and strict_chunking.

Rewrite a follow-up before searching, then request an answer with citations:

query = client.rewrite_query(
    "When?",
    generative_model="my-model",
    historical_messages=[{"user": "Who founded Acme?", "assistant": "Ada."}],
)
found = client.search("my_docs", query)
answer = client.generative_answer_with_citations(
    found.query_response_id,
    GenerativeOptions(generative_model="my-model", percentage_rank_mass=40),
)
print(answer.answer, answer.citations)

rewrite_query() returns a string and does not search or save a conversation. History turns are chronological and contain exactly user and assistant strings; the client's language is forwarded to SSE. This method requires the SSE rewrite_query endpoint from this release, not just an updated router. generative_answer_with_citations() accepts the same arguments as generate_answer() and always enables numbered sources, even when options explicitly disable them. It returns an Answer with .citations and delegates through SSE to the router's dedicated citation endpoint; it does not call the router directly or accept arbitrary document fragments.

The answer of GET collections also carries the counters of the collection: document_count (every document of the collection) and active_document_count (the documents which take part in the search, use_in_search=True). A Collection exposes both — None when the server did not send them — and document_count_or_zero / active_document_count_or_zero turn the missing counters into 0.

GET documents has two views. The default one lists the names of the documents which take part in the search (list_documents()); with with_disabled=True — and always in list_document_details() — the server adds the deactivated documents together with their id and use_in_search flag. Those identifiers are what document_ids[] of delete_documents and set_documents_use_in_search expects.

send_chat_message(..., collection="my_docs") requires an explicit collection, even if the chat was created with one: the current server resolves the collection from each message request. Omitting it raises SSEValueError before sending.

Set api_prefix="" for deployments without an API path prefix; an explicitly provided empty prefix overrides SSE_API_PREFIX.

Options objects

Class	Used by	Notes
SearchOptions	search()	Only the fields that are set are sent; extra merges unknown keys
IndexingOptions	upload_files(), index_documents()	Serialises the five cleaning keys the server reads with options["..."]; the two chunk keys and delete_files_after_indexing appear only when set, so the collection and the engine decide
GenerativeOptions	generate_answer(), send_chat_message()	None fields are dropped; carries use_doc_names_in_response, the flag the generator reads from the same query_options
ChatOptions	new_chat(), send_chat_message() (chat_options=)	The pipeline switches (use_rag_supervisor, use_content_supervisor, use_intelligent_query_rewrite, query_instruction, user_msg_is_rag_question) and the RAG memory settings (system_prompt_with_enumerated_documents, query_rewrite_model, rag_history_max_turns, rag_history_max_chars, history_only) merged into the single options object the chat endpoints read

Single options may also be passed as keyword arguments — the two calls below are equivalent:

client.search("my_docs", "query", SearchOptions(max_results=20, hybrid_search=True))
client.search("my_docs", "query", max_results=20, hybrid_search=True)

Unknown keys in search and generation dictionaries are preserved in extra; unknown indexing keys raise SSEValueError. Per-call dictionaries override only the supplied client defaults, while IndexingOptions instances supply a complete configuration (including their dataclass defaults). Keyword arguments win over both.

Wire-format quirks handled by the client

The backend is not uniform, and these traps are the reason the wrapper exists:

• options of search_with_options, query_options of generative_answer and indexing_options of upload_and_index_files are read with json.loads(), so they are sent as JSON strings;
• indexing_options of add_and_index_texts and options/search_options of the chat endpoints are read as objects, so they are sent as JSON objects;
• uploaded files use the field name files[] and input texts texts[];
• endpoint paths have no trailing slash (that is how Django registers them);
• business errors come back as HTTP 200 with {"status": false, "errors": [...]}.

Return values

Methods return dataclasses (sse_lib.models): Collection, SearchResponse (with .hits of SearchHit), Answer, UploadResult, IndexingResult, DocumentRef, Chat, ChatReply, ChatMessage, ChatHistory. Every one of them keeps the untouched server payload in .raw, and client.request("GET", "some_endpoint") returns the unwrapped body of any endpoint for fields that are not modelled yet.

The places where a citation leads back to the text are modelled directly: SearchHit.text_id / .document_id and .char_start / .char_end (the position of the chunk in the preview page, None when it cannot be reproduced), DocumentPreview.highlight for the page returned by document_content(text_id=...), Answer.citations for the numbered sources of an answer, and ChatReply.citations / .rewritten_query / .answer_source ("rag", "history" or "llm_fallback") for a reply of a chat.

Errors

Exception	Raised when
SSEValueError	An argument is rejected before the request is sent (empty query, unknown index type, ...)
SSEConfigError	Invalid configuration (missing host, non-positive timeout, login without credentials)
SSEAPIError	The API answered {"status": false, "errors": [...]} — inspect .error_names, .not_given_params
SSEAuthenticationError	HTTP 401/403
SSEHTTPError	Any other non-2xx status
SSETransportError	No response at all (DNS, connection refused, timeout)

All of them derive from SSEError, so a single except SSEError is enough for a coarse handler. Idempotent GET calls are retried on connection errors and on 429/502/503/504; POST is never replayed, because indexing twice is a real side effect.

Development

cd sse_lib
PYTHONPATH=src python -m unittest discover -s tests -v   # no installation needed
python -m pytest .                                        # pytest, if available
black --check . && flake8 .

The tests never open a socket: tests/support.py replaces the HTTP session with a recording double, which pins the exact payloads sent to the API.