Railengine Retrieval SDK
Python SDK for retrieving and searching data from Railtown AI Railengine - handles embeddings, storage documents, and indexed content.
Overview
The rail-engine package provides a Pythonic interface for retrieving and searching data from Railengine. It supports async/await patterns, client-side filtering, explicit storage list paging via StorageListPage, and Pydantic model deserialization.
Installation
uv pip install rail-engine
Quick Start
import asyncio
from railtown.engine import Railengine
async def main():
# Initialize client (reads from ENGINE_PAT and ENGINE_ID env vars)
async with Railengine() as client:
page = await client.list_storage_documents(
page_number=1,
page_size=25,
)
print(f"total_count={page.total_count} total_pages={page.total_pages}")
for doc in page.items:
print(doc)
asyncio.run(main())
Configuration
Environment Variables
ENGINE_PAT(required) - Personal Access TokenENGINE_ID(required) - Engine ID (can also be passed to constructor)RAILTOWN_API_URL(optional) - Base API URL (defaults tohttps://cndr.railtown.ai/api)
Constructor Parameters
pat(optional) - PAT token (if not provided, reads fromENGINE_PATenv)engine_id(optional) - Engine ID (if not provided, reads fromENGINE_IDenv, required if not in env)api_url(optional) - Base API URL (if not provided, reads fromRAILTOWN_API_URLenv or defaults to production)model(optional) - Pydantic model type for deserializing retrieved datatimeout(optional) - Request timeout in seconds orhttpx.Timeout(default30). Applied to the SDK-created client and to each request, including vector search.verify(optional) - TLS verification. Defaults toTrue. Pass a CA bundle path orssl.SSLContextfor custom trust.Falsedisables verification and must be set explicitly.http_client(optional) - Caller-ownedhttpx.AsyncClientfor proxies, custom transports, or tests. The SDK does not close this client.retrieval_error_mode(optional) -"raise"(default) or"fallback"; see the error contract below.logging_policy(optional) -LoggingPolicyfor SDK logs. Default logs operation metadata only. Search text, JSONPath, and response bodies are not logged unlesslog_content=True.
Earlier SDK versions disabled TLS verification. 0.4.0 verifies certificates by default. Use verify= with your CA bundle if the API host uses a private certificate.
Logging and sensitive data
SDK loggers use the railtown.engine.* namespace. The host application controls logging handlers and levels.
- With the default
LoggingPolicy(log_content=False), SDK logs contain operation metadata such as engine id, status, and duration. Queries, request payloads, and response bodies are excluded, including at DEBUG. Raising the log level alone does not enable content logging. - Explicitly setting
LoggingPolicy(log_content=True)allows queries and request/response content in SDK logs. Request content can appear at INFO, and request/response content can appear at ERROR on failures; content logging is not restricted to DEBUG. - Opt-in content logging applies secret redaction and truncates content to 512 characters by default. A custom
redactorandcontent_max_charscan adjust this behavior. Redaction and truncation do not guarantee that all sensitive data is removed. Leavelog_content=Falsein production. httpxlogs independently of the SDK policy. Its INFO logs can include full URLs with query strings, which may contain sensitive queries. Set thehttpxlogger to WARNING in production:
import logging
logging.getLogger("httpx").setLevel(logging.WARNING)
Features
- Multiple retrieval methods:
search_vector_store()- Semantic search in vector storessearch_vector_store_filtered()- Semantic search restricted to the documents an index query matches (search_querytakes the sameIndexingSearchQuery/dict assearch_index; requires Indexing and Embeddings enabled on the engine)get_storage_document_by_event_id()- Get document by EventIdget_storage_document_by_customer_key()- Same paging aslist_storage_documents(page_number,page_size) but scoped to a CustomerKey; returnsStorageListPagewithitems,total_pages, andtotal_countquery_storage_by_jsonpath()- Query using JSONPathlist_storage_documents()- Usepage_numberandpage_sizeto paginate; returnsStorageListPagewithitems,total_pages, andtotal_countsearch_index()- Index search (defaultraw=True: each hit is the API dict; useraw=Falseto parse each hit'sbodywithmodelor the client default); returnsIndexingSearchResultwithitemsand optionaltotal_countdelete_document()- Delete all document versions by customer key from storage, embeddings, and indexing (raises on API errors)
- Client-side filtering on storage reads - Optional
filter_fnon storage/list/query helpers (not onsearch_vector_storeorsearch_vector_store_filtered; filter vector hits in your loop) - Storage list pagination - One HTTP GET per page; use
page_number/page_sizeon the request andtotal_pages/total_countonStorageListPagewhen looping - Model deserialization - Optional Pydantic model support with per-call override
- Actionable error handling - Retrieval failures raise by default; explicit fallback mode supports legacy integrations
- Async/await support - Built for modern async Python applications
Vector search: document models and hit metadata
Migration to 0.6.1
Version 0.6.1 intentionally changes both vector search methods. Every normal result
is a VectorSearchResult: result.hit contains the outer search metadata and
result.document contains the separately validated document. Document fields can
never replace the hit identifier, score, or other metadata.
from pydantic import BaseModel
from railtown.engine import Railengine, RailtownSearchParsingError
class Food(BaseModel):
name: str
async def search_documents():
async with Railengine(model=Food) as client:
try:
results = await client.search_vector_store(query="apple")
for result in results:
assert result.document is not None
print(result.hit.event_id, result.hit.score, result.document.name)
except RailtownSearchParsingError as error:
# Zero-based response row, outer event ID, metadata/document phase,
# and a fixed failure category (no document values or field paths).
print(error.row_index, error.event_id, error.phase, error.problem)
A per-call model overrides the client default. Omitting it or passing model=None
keeps the default. Models validate only the JSON in hit.content; there is no
whole-hit fallback. Invalid JSON or a model mismatch raises
RailtownSearchParsingError, including when retrieval_error_mode="fallback".
One invalid hit fails the call: no partial list, skipped hit, or mixed dictionary
fallback is returned. Invalid outer hit metadata also raises this error.
Without any model, results still have typed hit metadata; document is None
and the original content remains available as result.hit.content. Pass a domain
model to parse it. Passing VectorStoreSearchHit as the document model only parses
a nested document of that shape; it never requests or replaces outer metadata.
To request unparsed hit dictionaries, explicitly pass raw=True. This bypasses
both per-call and client models, including document and hit-field validation.
Response envelope validation and HTTP error handling still apply.
async def search_raw():
async with Railengine(model=Food) as client:
rows = await client.search_vector_store(query="apple", raw=True)
for row in rows:
print(row["eventId"], row["score"], row["content"])
Upgrade existing callers as follows:
- Replace dictionary access such as
row["eventId"]withresult.hit.event_id, or explicitly selectraw=Trueto retain dictionary access. - Replace direct document access such as
item.namewithresult.document.name. - Replace
model=VectorStoreSearchHitused for metadata with no document model and readresult.hit. With a configured default, use raw mode if parsing is unwanted. - Catch
RailtownSearchParsingErrorand correct invalid documents or choose raw mode deliberately. Do not rely on mixed result types or whole-hit fallback. - Apply the same migration to
search_vector_store_filtered; itssearch_queryand ranking behavior are unchanged.
This patch release includes breaking vector search changes. Upgrade and test callers
before moving to 0.6.1; pin <0.6.1 temporarily if migration must wait.
There is no implicit legacy parsing mode. Storage and index parsing contracts are unchanged.
Error Handling
The SDK provides custom exception classes:
RailtownSearchParsingError- Invalid vector hit metadata or document contentRailtownError- Base exceptionRailtownBadRequestError- 400 Bad RequestRailtownUnauthorizedError- 401 UnauthorizedRailtownNotFoundError- 404 Not FoundRailtownConflictError- 409 ConflictRailtownServerError- 5xx Server errors
Error contract and migration to 0.5.0
Version 0.5.0 changes retrieval failures to raise by default. Successful empty queries keep their existing result types and pagination metadata. To retain the previous retrieval fallback behavior, explicitly construct Railengine(retrieval_error_mode="fallback"). Fallback logs the failure and returns an empty list, empty result/page, or None for an event lookup. It does not suppress ingestion or deletion failures, invalid arguments, cancellation, or application callback errors. Storage and index model parsing are unchanged. Vector search uses the strict 0.6.1 contract above.
A 404 raises RailtownNotFoundError by default, including event lookups. The current API contract does not establish a reliable discriminator between a missing document and another missing resource; the SDK does not infer which one is absent or perform another request to guess.
Catch the common base class for all request/response errors:
from railtown.engine import Railengine, RailtownError
async def search():
async with Railengine() as client:
try:
return await client.search_index(query={"search": "example"})
except RailtownError as error:
print(error.operation, error.status_code, error.request_id)
raise
HTTP 400, 401, 403, 404, 409, and 429 map to RailtownBadRequestError, RailtownUnauthorizedError, RailtownForbiddenError, RailtownNotFoundError, RailtownConflictError, and RailtownRateLimitError. Other 4xx responses use RailtownError; 5xx responses use RailtownServerError. Transport failures use RailtownTransportError, timeouts its subclass RailtownTimeoutError, and invalid response JSON/envelopes use RailtownResponseError. For retrieval (rail-engine), import these exceptions from railtown.engine. For ingestion (rail-engine-ingest), import them from railtown.engine.ingest. The railtown.engine.ingest namespace requires rail-engine-ingest; it is not included in the retrieval wheel.
Request/response errors expose status_code when available, operation, the effective engine_id, request_id when available, and retryable. Retryable is true for timeouts, connection failures, and HTTP 408, 429, 502, 503, and 504. It describes a potentially transient failure, not whether replaying a write is safe. The SDK does not retry automatically.
Diagnostic compatibility change: SDK-generated errors now leave response_text as None. The constructor still accepts an explicitly supplied value. Messages exclude query strings and payloads, but the preserved original exception cause can retain sensitive data. Full traceback logging is not guaranteed to be sanitized. Explicit content logging remains governed by LoggingPolicy and does not populate error attributes.
Requirements
- uv
- Python 3.11+
- httpx >= 0.24.0
- pydantic >= 1.10.0
Related package
For ingesting data into Railengine, install and use rail-engine-ingest.
License
MIT
Document deletion migration (0.6.0)
delete_event has been removed. Use await client.delete_document(customer_key, engine_id=None) with the original customer key, not an event ID. This deletes all existing versions. Requires the updated Document API and a ProjectReadAdmin PAT. Coordinate API and SDK releases.
202 indicates an index deletion was accepted. 204 indicates storage deletion succeeded but may include index failure. Index cleanup is best effort and eventually consistent. In-flight processing or later ingestion can recreate data. API and transport errors raise SDK exceptions.
Metadata
Release files for rail-engine 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| rail_engine-0.6.1.tar.gz | 32.5 kB | Without provenance | Details for rail_engine-0.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| rail_engine-0.6.1-py3-none-any.whl | Python 3 | none | any | Details for rail_engine-0.6.1-py3-none-any.whl |
Total release size: 66.0 kB
Release files / rail_engine-0.6.1.tar.gz
| Download URL | rail_engine-0.6.1.tar.gz |
|---|---|
| Size | 32.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
88d4cd19f7b10a15b8b2c63dcc5074cef96a8da307ee159abbc9cb2582c4f32d
|
|
BLAKE2b-256 checksum How to use checksums |
c068ac5bc4da6a2936ce7ee8f7b6bb1902c8ad5ad02c34c4cdf96f998617720b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / rail_engine-0.6.1-py3-none-any.whl
| Download URL | rail_engine-0.6.1-py3-none-any.whl |
|---|---|
| Size | 33.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4546a47245af313c9ec96cdb3482ee3a5fe27f8d4a100198ef3caf062a8bd79f
|
|
BLAKE2b-256 checksum How to use checksums |
c51bb29bfd6c843f0f402d84ba3bd994c1a383504211a0c64405f9d232486550
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for rail-engine 0.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.