Skip to main content

Railengine Retrieval SDK

Python SDK for retrieving and searching data from Railtown AI Railengine - handles embeddings, storage documents, and indexed content.

Overview

The rail-engine package provides a Pythonic interface for retrieving and searching data from Railengine. It supports async/await patterns, client-side filtering, explicit storage list paging via StorageListPage, and Pydantic model deserialization.

Installation

uv pip install rail-engine

Quick Start

import asyncio
from railtown.engine import Railengine

async def main():
    # Initialize client (reads from ENGINE_PAT and ENGINE_ID env vars)
    async with Railengine() as client:
        page = await client.list_storage_documents(
            page_number=1,
            page_size=25,
        )
        print(f"total_count={page.total_count} total_pages={page.total_pages}")
        for doc in page.items:
            print(doc)

asyncio.run(main())

Configuration

Environment Variables

  • ENGINE_PAT (required) - Personal Access Token
  • ENGINE_ID (required) - Engine ID (can also be passed to constructor)
  • RAILTOWN_API_URL (optional) - Base API URL (defaults to https://cndr.railtown.ai/api)

Constructor Parameters

  • pat (optional) - PAT token (if not provided, reads from ENGINE_PAT env)
  • engine_id (optional) - Engine ID (if not provided, reads from ENGINE_ID env, required if not in env)
  • api_url (optional) - Base API URL (if not provided, reads from RAILTOWN_API_URL env or defaults to production)
  • model (optional) - Pydantic model type for deserializing retrieved data
  • timeout (optional) - Request timeout in seconds or httpx.Timeout (default 30). Applied to the SDK-created client and to each request, including vector search.
  • verify (optional) - TLS verification. Defaults to True. Pass a CA bundle path or ssl.SSLContext for custom trust. False disables verification and must be set explicitly.
  • http_client (optional) - Caller-owned httpx.AsyncClient for proxies, custom transports, or tests. The SDK does not close this client.
  • retrieval_error_mode (optional) - "raise" (default) or "fallback"; see the error contract below.
  • logging_policy (optional) - LoggingPolicy for SDK logs. Default logs operation metadata only. Search text, JSONPath, and response bodies are not logged unless log_content=True.

Earlier SDK versions disabled TLS verification. 0.4.0 verifies certificates by default. Use verify= with your CA bundle if the API host uses a private certificate.

Logging and sensitive data

SDK loggers use the railtown.engine.* namespace. The host application controls logging handlers and levels.

  • With the default LoggingPolicy(log_content=False), SDK logs contain operation metadata such as engine id, status, and duration. Queries, request payloads, and response bodies are excluded, including at DEBUG. Raising the log level alone does not enable content logging.
  • Explicitly setting LoggingPolicy(log_content=True) allows queries and request/response content in SDK logs. Request content can appear at INFO, and request/response content can appear at ERROR on failures; content logging is not restricted to DEBUG.
  • Opt-in content logging applies secret redaction and truncates content to 512 characters by default. A custom redactor and content_max_chars can adjust this behavior. Redaction and truncation do not guarantee that all sensitive data is removed. Leave log_content=False in production.
  • httpx logs independently of the SDK policy. Its INFO logs can include full URLs with query strings, which may contain sensitive queries. Set the httpx logger to WARNING in production:
import logging

logging.getLogger("httpx").setLevel(logging.WARNING)

Features

  • Multiple retrieval methods:
    • search_vector_store() - Semantic search in vector stores
    • search_vector_store_filtered() - Semantic search restricted to the documents an index query matches (search_query takes the same IndexingSearchQuery/dict as search_index; requires Indexing and Embeddings enabled on the engine)
    • get_storage_document_by_event_id() - Get document by EventId
    • get_storage_document_by_customer_key() - Same paging as list_storage_documents (page_number, page_size) but scoped to a CustomerKey; returns StorageListPage with items, total_pages, and total_count
    • query_storage_by_jsonpath() - Query using JSONPath
    • list_storage_documents() - Use page_number and page_size to paginate; returns StorageListPage with items, total_pages, and total_count
    • search_index() - Index search (default raw=True: each hit is the API dict; use raw=False to parse each hit's body with model or the client default); returns IndexingSearchResult with items and optional total_count
    • delete_document() - Delete all document versions by customer key from storage, embeddings, and indexing (raises on API errors)
  • Client-side filtering on storage reads - Optional filter_fn on storage/list/query helpers (not on search_vector_store or search_vector_store_filtered; filter vector hits in your loop)
  • Storage list pagination - One HTTP GET per page; use page_number / page_size on the request and total_pages / total_count on StorageListPage when looping
  • Model deserialization - Optional Pydantic model support with per-call override
  • Actionable error handling - Retrieval failures raise by default; explicit fallback mode supports legacy integrations
  • Async/await support - Built for modern async Python applications

Vector search: document models and hit metadata

Migration to 0.6.1

Version 0.6.1 intentionally changes both vector search methods. Every normal result is a VectorSearchResult: result.hit contains the outer search metadata and result.document contains the separately validated document. Document fields can never replace the hit identifier, score, or other metadata.

from pydantic import BaseModel
from railtown.engine import Railengine, RailtownSearchParsingError

class Food(BaseModel):
    name: str

async def search_documents():
    async with Railengine(model=Food) as client:
        try:
            results = await client.search_vector_store(query="apple")
            for result in results:
                assert result.document is not None
                print(result.hit.event_id, result.hit.score, result.document.name)
        except RailtownSearchParsingError as error:
            # Zero-based response row, outer event ID, metadata/document phase,
            # and a fixed failure category (no document values or field paths).
            print(error.row_index, error.event_id, error.phase, error.problem)

A per-call model overrides the client default. Omitting it or passing model=None keeps the default. Models validate only the JSON in hit.content; there is no whole-hit fallback. Invalid JSON or a model mismatch raises RailtownSearchParsingError, including when retrieval_error_mode="fallback". One invalid hit fails the call: no partial list, skipped hit, or mixed dictionary fallback is returned. Invalid outer hit metadata also raises this error.

Without any model, results still have typed hit metadata; document is None and the original content remains available as result.hit.content. Pass a domain model to parse it. Passing VectorStoreSearchHit as the document model only parses a nested document of that shape; it never requests or replaces outer metadata.

To request unparsed hit dictionaries, explicitly pass raw=True. This bypasses both per-call and client models, including document and hit-field validation. Response envelope validation and HTTP error handling still apply.

async def search_raw():
    async with Railengine(model=Food) as client:
        rows = await client.search_vector_store(query="apple", raw=True)
        for row in rows:
            print(row["eventId"], row["score"], row["content"])

Upgrade existing callers as follows:

  • Replace dictionary access such as row["eventId"] with result.hit.event_id, or explicitly select raw=True to retain dictionary access.
  • Replace direct document access such as item.name with result.document.name.
  • Replace model=VectorStoreSearchHit used for metadata with no document model and read result.hit. With a configured default, use raw mode if parsing is unwanted.
  • Catch RailtownSearchParsingError and correct invalid documents or choose raw mode deliberately. Do not rely on mixed result types or whole-hit fallback.
  • Apply the same migration to search_vector_store_filtered; its search_query and ranking behavior are unchanged.

This patch release includes breaking vector search changes. Upgrade and test callers before moving to 0.6.1; pin <0.6.1 temporarily if migration must wait. There is no implicit legacy parsing mode. Storage and index parsing contracts are unchanged.

Error Handling

The SDK provides custom exception classes:

  • RailtownSearchParsingError - Invalid vector hit metadata or document content
  • RailtownError - Base exception
  • RailtownBadRequestError - 400 Bad Request
  • RailtownUnauthorizedError - 401 Unauthorized
  • RailtownNotFoundError - 404 Not Found
  • RailtownConflictError - 409 Conflict
  • RailtownServerError - 5xx Server errors

Error contract and migration to 0.5.0

Version 0.5.0 changes retrieval failures to raise by default. Successful empty queries keep their existing result types and pagination metadata. To retain the previous retrieval fallback behavior, explicitly construct Railengine(retrieval_error_mode="fallback"). Fallback logs the failure and returns an empty list, empty result/page, or None for an event lookup. It does not suppress ingestion or deletion failures, invalid arguments, cancellation, or application callback errors. Storage and index model parsing are unchanged. Vector search uses the strict 0.6.1 contract above.

A 404 raises RailtownNotFoundError by default, including event lookups. The current API contract does not establish a reliable discriminator between a missing document and another missing resource; the SDK does not infer which one is absent or perform another request to guess.

Catch the common base class for all request/response errors:

from railtown.engine import Railengine, RailtownError

async def search():
    async with Railengine() as client:
        try:
            return await client.search_index(query={"search": "example"})
        except RailtownError as error:
            print(error.operation, error.status_code, error.request_id)
            raise

HTTP 400, 401, 403, 404, 409, and 429 map to RailtownBadRequestError, RailtownUnauthorizedError, RailtownForbiddenError, RailtownNotFoundError, RailtownConflictError, and RailtownRateLimitError. Other 4xx responses use RailtownError; 5xx responses use RailtownServerError. Transport failures use RailtownTransportError, timeouts its subclass RailtownTimeoutError, and invalid response JSON/envelopes use RailtownResponseError. For retrieval (rail-engine), import these exceptions from railtown.engine. For ingestion (rail-engine-ingest), import them from railtown.engine.ingest. The railtown.engine.ingest namespace requires rail-engine-ingest; it is not included in the retrieval wheel.

Request/response errors expose status_code when available, operation, the effective engine_id, request_id when available, and retryable. Retryable is true for timeouts, connection failures, and HTTP 408, 429, 502, 503, and 504. It describes a potentially transient failure, not whether replaying a write is safe. The SDK does not retry automatically.

Diagnostic compatibility change: SDK-generated errors now leave response_text as None. The constructor still accepts an explicitly supplied value. Messages exclude query strings and payloads, but the preserved original exception cause can retain sensitive data. Full traceback logging is not guaranteed to be sanitized. Explicit content logging remains governed by LoggingPolicy and does not populate error attributes.

Requirements

  • uv
  • Python 3.11+
  • httpx >= 0.24.0
  • pydantic >= 1.10.0

For ingesting data into Railengine, install and use rail-engine-ingest.

License

MIT

Document deletion migration (0.6.0)

delete_event has been removed. Use await client.delete_document(customer_key, engine_id=None) with the original customer key, not an event ID. This deletes all existing versions. Requires the updated Document API and a ProjectReadAdmin PAT. Coordinate API and SDK releases.

202 indicates an index deletion was accepted. 204 indicates storage deletion succeeded but may include index failure. Index cleanup is best effort and eventually consistent. In-flight processing or later ingestion can recreate data. API and transport errors raise SDK exceptions.

Metadata

Release files for rail-engine 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rail-engine 0.6.1
File Size Uploaded Provenance Actions
rail_engine-0.6.1.tar.gz 32.5 kB Without provenance Details for rail_engine-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rail-engine 0.6.1
File Interpreter ABI Platform Actions
rail_engine-0.6.1-py3-none-any.whl Python 3 none any Details for rail_engine-0.6.1-py3-none-any.whl

Total release size: 66.0 kB

Release files / rail_engine-0.6.1.tar.gz

Download URL rail_engine-0.6.1.tar.gz
Size 32.5 kB
Tags Source
SHA-256 checksum
How to use checksums
88d4cd19f7b10a15b8b2c63dcc5074cef96a8da307ee159abbc9cb2582c4f32d
BLAKE2b-256 checksum
How to use checksums
c068ac5bc4da6a2936ce7ee8f7b6bb1902c8ad5ad02c34c4cdf96f998617720b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / rail_engine-0.6.1-py3-none-any.whl

Download URL rail_engine-0.6.1-py3-none-any.whl
Size 33.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4546a47245af313c9ec96cdb3482ee3a5fe27f8d4a100198ef3caf062a8bd79f
BLAKE2b-256 checksum
How to use checksums
c51bb29bfd6c843f0f402d84ba3bd994c1a383504211a0c64405f9d232486550
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for rail-engine 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page