Railengine Ingestion SDK
Python SDK for ingesting data into Railtown AI Railengine - handles embeddings, storage, indexing, and webhook publishing.
Overview
The rail-engine-ingest package provides a Pythonic interface for publishing data to Railengine. It supports async/await patterns, Pydantic model validation, and familiar configuration patterns.
Installation
uv pip install rail-engine-ingest
Quick Start
# Copy your ENGINE_TOKEN from https://cndr.railtown.ai Project Settings
import asyncio
import os
from railtown.engine.ingest import RailengineIngest
import base64
import json
engine_token = os.getenv("ENGINE_TOKEN")
async def main():
# Initialize client
async with RailengineIngest(engine_token=engine_token) as client:
# Ingest data
data = {
"name": "Chicken Shawarma",
"meal": "lunch",
"calories": 95
}
response = await client.upsert(data)
print(f"Status: {response.status_code}")
asyncio.run(main())
Configuration
Environment Variable
ENGINE_TOKEN- Base64-encoded JSON string containing ingestion credentials
Constructor Parameters
engine_token(optional) - ENGINE_TOKEN string (if not provided, reads from env)model(optional) - Pydantic model type for validating ingested datatimeout(optional) - Request timeout in seconds orhttpx.Timeout(default30)verify(optional) - TLS verification. Defaults toTrue. Pass a CA bundle path orssl.SSLContextfor custom trust.Falsedisables verification and must be set explicitly.http_client(optional) - Caller-ownedhttpx.AsyncClientfor proxies, custom transports, or tests. The SDK does not close this client.logging_policy(optional) -LoggingPolicyfor SDK logs. Default INFO logs include engine id, status, duration, and payload size. The request body is not logged unlesslog_content=True.
Earlier SDK versions disabled TLS verification. 0.4.0 verifies certificates by default. Use verify= with your CA bundle if the ingestion host uses a private certificate.
Logging and sensitive data
SDK loggers use the railtown.engine.* namespace. The host application controls logging handlers and levels.
- With the default
LoggingPolicy(log_content=False), SDK logs contain operation metadata such as engine id, status, and duration. Queries, request payloads, and response bodies are excluded, including at DEBUG. Raising the log level alone does not enable content logging. - Explicitly setting
LoggingPolicy(log_content=True)allows queries and request/response content in SDK logs. Request content can appear at INFO, and request/response content can appear at ERROR on failures; content logging is not restricted to DEBUG. - Opt-in content logging applies secret redaction and truncates content to 512 characters by default. A custom
redactorandcontent_max_charscan adjust this behavior. Redaction and truncation do not guarantee that all sensitive data is removed. Leavelog_content=Falsein production. httpxlogs independently of the SDK policy. Its INFO logs can include full URLs with query strings, which may contain sensitive queries. Set thehttpxlogger to WARNING in production:
import logging
logging.getLogger("httpx").setLevel(logging.WARNING)
Features
- Single
upsert()method - Unified interface for ingesting data - Multiple data formats - Accepts Pydantic models, dictionaries, or JSON strings
- Model validation - Optional Pydantic model validation
- Direct JSON payload - Sends data directly as JSON to the ingestion endpoint
- Webhook handler -
get_webhook_handler()returns a payload parser when the client is constructed withmodel= - Async/await support - Built for modern async Python applications
Error Handling
The SDK provides custom exception classes:
RailtownError- Base exceptionRailtownBadRequestError- 400 Bad RequestRailtownUnauthorizedError- 401 UnauthorizedRailtownNotFoundError- 404 Not FoundRailtownConflictError- 409 ConflictRailtownServerError- 5xx Server errors
Exceptions are raised on errors. In 0.5.0, ingestion shares the retrieval error classification: HTTP 403 and 429 use RailtownForbiddenError and RailtownRateLimitError, transport failures use RailtownTransportError, and timeouts use its subclass RailtownTimeoutError. HTTP 404 and 409 use RailtownNotFoundError and RailtownConflictError. These types are exported from railtown.engine.ingest; all inherit from RailtownError.
Request errors expose status_code when available, operation, engine_id, request_id when available, and retryable. Retryable is true for timeouts, connection failures, and HTTP 408, 429, 502, 503, and 504. It does not mean replaying an ingestion request is safe, and no automatic retries are performed.
Migration to 0.5.0: SDK-generated errors leave response_text as None; existing exception constructors still accept explicitly supplied values. SDK-generated messages exclude request/response payloads, but the preserved original cause can retain sensitive data. Full traceback logging is not guaranteed to be sanitized. Ingestion always raises on request failures, independent of the retrieval compatibility mode. User-model validation behavior is unchanged.
Requirements
- uv
- Python 3.11+
- httpx >= 0.24.0
- pydantic >= 1.10.0
Related package
For retrieving and searching data from Railengine, install and use rail-engine.
License
MIT
Metadata
Release files for rail-engine-ingest 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| rail_engine_ingest-0.6.1.tar.gz | 36.1 kB | Without provenance | Details for rail_engine_ingest-0.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| rail_engine_ingest-0.6.1-py3-none-any.whl | Python 3 | none | any | Details for rail_engine_ingest-0.6.1-py3-none-any.whl |
Total release size: 73.4 kB
Release files / rail_engine_ingest-0.6.1.tar.gz
| Download URL | rail_engine_ingest-0.6.1.tar.gz |
|---|---|
| Size | 36.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d11b06f50be16eec2d7c5b86da3ff48d442a2d79381a28e67c0478ce7a078ac4
|
|
BLAKE2b-256 checksum How to use checksums |
66df3eb92ca10462ff4fb280b70aaaa6b1ee1fd10bc484859f332137d666601e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / rail_engine_ingest-0.6.1-py3-none-any.whl
| Download URL | rail_engine_ingest-0.6.1-py3-none-any.whl |
|---|---|
| Size | 37.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9c1dbc0bba09fb0edc8c2d059da839200df447fdcbd6a4ad7fecb13a68e09054
|
|
BLAKE2b-256 checksum How to use checksums |
94557a4fc8fc986a9bf045185e5309f020c99161b8bb272d6a519fe9e2e754ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for rail-engine-ingest 0.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.