Skip to main content

Railengine Ingestion SDK

Python SDK for ingesting data into Railtown AI Railengine - handles embeddings, storage, indexing, and webhook publishing.

Overview

The rail-engine-ingest package provides a Pythonic interface for publishing data to Railengine. It supports async/await patterns, Pydantic model validation, and familiar configuration patterns.

Installation

uv pip install rail-engine-ingest

Quick Start

# Copy your ENGINE_TOKEN from https://cndr.railtown.ai Project Settings

import asyncio
import os
from railtown.engine.ingest import RailengineIngest
import base64
import json

engine_token = os.getenv("ENGINE_TOKEN")

async def main():
    # Initialize client
    async with RailengineIngest(engine_token=engine_token) as client:
        # Ingest data
        data = {
            "name": "Chicken Shawarma",
            "meal": "lunch",
            "calories": 95
        }
        response = await client.upsert(data)
        print(f"Status: {response.status_code}")

asyncio.run(main())

Configuration

Environment Variable

  • ENGINE_TOKEN - Base64-encoded JSON string containing ingestion credentials

Constructor Parameters

  • engine_token (optional) - ENGINE_TOKEN string (if not provided, reads from env)
  • model (optional) - Pydantic model type for validating ingested data
  • timeout (optional) - Request timeout in seconds or httpx.Timeout (default 30)
  • verify (optional) - TLS verification. Defaults to True. Pass a CA bundle path or ssl.SSLContext for custom trust. False disables verification and must be set explicitly.
  • http_client (optional) - Caller-owned httpx.AsyncClient for proxies, custom transports, or tests. The SDK does not close this client.
  • logging_policy (optional) - LoggingPolicy for SDK logs. Default INFO logs include engine id, status, duration, and payload size. The request body is not logged unless log_content=True.

Earlier SDK versions disabled TLS verification. 0.4.0 verifies certificates by default. Use verify= with your CA bundle if the ingestion host uses a private certificate.

Logging and sensitive data

SDK loggers use the railtown.engine.* namespace. The host application controls logging handlers and levels.

  • With the default LoggingPolicy(log_content=False), SDK logs contain operation metadata such as engine id, status, and duration. Queries, request payloads, and response bodies are excluded, including at DEBUG. Raising the log level alone does not enable content logging.
  • Explicitly setting LoggingPolicy(log_content=True) allows queries and request/response content in SDK logs. Request content can appear at INFO, and request/response content can appear at ERROR on failures; content logging is not restricted to DEBUG.
  • Opt-in content logging applies secret redaction and truncates content to 512 characters by default. A custom redactor and content_max_chars can adjust this behavior. Redaction and truncation do not guarantee that all sensitive data is removed. Leave log_content=False in production.
  • httpx logs independently of the SDK policy. Its INFO logs can include full URLs with query strings, which may contain sensitive queries. Set the httpx logger to WARNING in production:
import logging

logging.getLogger("httpx").setLevel(logging.WARNING)

Features

  • Single upsert() method - Unified interface for ingesting data
  • Multiple data formats - Accepts Pydantic models, dictionaries, or JSON strings
  • Model validation - Optional Pydantic model validation
  • Direct JSON payload - Sends data directly as JSON to the ingestion endpoint
  • Webhook handler - get_webhook_handler() returns a payload parser when the client is constructed with model=
  • Async/await support - Built for modern async Python applications

Error Handling

The SDK provides custom exception classes:

  • RailtownError - Base exception
  • RailtownBadRequestError - 400 Bad Request
  • RailtownUnauthorizedError - 401 Unauthorized
  • RailtownNotFoundError - 404 Not Found
  • RailtownConflictError - 409 Conflict
  • RailtownServerError - 5xx Server errors

Exceptions are raised on errors. In 0.5.0, ingestion shares the retrieval error classification: HTTP 403 and 429 use RailtownForbiddenError and RailtownRateLimitError, transport failures use RailtownTransportError, and timeouts use its subclass RailtownTimeoutError. HTTP 404 and 409 use RailtownNotFoundError and RailtownConflictError. These types are exported from railtown.engine.ingest; all inherit from RailtownError.

Request errors expose status_code when available, operation, engine_id, request_id when available, and retryable. Retryable is true for timeouts, connection failures, and HTTP 408, 429, 502, 503, and 504. It does not mean replaying an ingestion request is safe, and no automatic retries are performed.

Migration to 0.5.0: SDK-generated errors leave response_text as None; existing exception constructors still accept explicitly supplied values. SDK-generated messages exclude request/response payloads, but the preserved original cause can retain sensitive data. Full traceback logging is not guaranteed to be sanitized. Ingestion always raises on request failures, independent of the retrieval compatibility mode. User-model validation behavior is unchanged.

Requirements

  • uv
  • Python 3.11+
  • httpx >= 0.24.0
  • pydantic >= 1.10.0

For retrieving and searching data from Railengine, install and use rail-engine.

License

MIT

Metadata

Release files for rail-engine-ingest 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rail-engine-ingest 0.6.1
File Size Uploaded Provenance Actions
rail_engine_ingest-0.6.1.tar.gz 36.1 kB Without provenance Details for rail_engine_ingest-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rail-engine-ingest 0.6.1
File Interpreter ABI Platform Actions
rail_engine_ingest-0.6.1-py3-none-any.whl Python 3 none any Details for rail_engine_ingest-0.6.1-py3-none-any.whl

Total release size: 73.4 kB

Release files / rail_engine_ingest-0.6.1.tar.gz

Download URL rail_engine_ingest-0.6.1.tar.gz
Size 36.1 kB
Tags Source
SHA-256 checksum
How to use checksums
d11b06f50be16eec2d7c5b86da3ff48d442a2d79381a28e67c0478ce7a078ac4
BLAKE2b-256 checksum
How to use checksums
66df3eb92ca10462ff4fb280b70aaaa6b1ee1fd10bc484859f332137d666601e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / rail_engine_ingest-0.6.1-py3-none-any.whl

Download URL rail_engine_ingest-0.6.1-py3-none-any.whl
Size 37.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9c1dbc0bba09fb0edc8c2d059da839200df447fdcbd6a4ad7fecb13a68e09054
BLAKE2b-256 checksum
How to use checksums
94557a4fc8fc986a9bf045185e5309f020c99161b8bb272d6a519fe9e2e754ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for rail-engine-ingest 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page