Skip to main content

Haystack ML Stack

Currently this project contains a FastAPI-based service designed for low-latency scoring of streams data coming from http requests

🚀 Features

  • FastAPI Service: Lightweight and fast web service for ML inference.
    • Asynchronous I/O: Utilizes aiobotocore for non-blocking S3 and DynamoDB operations.
    • Model Loading: Downloads and loads the ML model (using cloudpickle) from a configurable S3 path on startup.
    • Feature Caching: Implements a thread-safe Time-To-Live (TTL) / Least-Recently-Used (LRU) cache (cachetools.TLRUCache) for DynamoDB features, reducing latency and database load.
    • DynamoDB Integration: Fetches stream-specific features from DynamoDB to enrich the data before scoring.
    • Health Check: Provides a /health endpoint to monitor service status and model loading.

📦 Installation

This project requires Python 3.11 or later.

  1. Install package: The dependencies associated are listed in pyproject.toml.

    pip install haystack-ml-stack
    

⚙️ Configuration

The service is configured using environment variables, managed by pydantic-settings. You can use a .env file for local development.

Variable Name Alias Default Description
S3_MODEL_PATH S3_MODEL_PATH None Required. The s3://bucket/key URL for the cloudpickled ML model file.
FEATURES_TABLE FEATURES_TABLE "features" Name of the DynamoDB table storing stream features.
LOGS_FRACTION LOGS_FRACTION 0.01 Fraction of requests to log detailed stream data for sampling/debugging (0.0 to 1.0).
CACHE_MAXSIZE (none) 50000 Maximum size of the in-memory feature cache.

Example env vars

S3_MODEL_PATH="s3://my-ml-models/stream-scorer/latest.pkl"
FEATURES_TABLE="features"
LOGS_FRACTION=0.05

🌐 Endpoints

Method Path Description
GET / Root endpoint, returns a simple running message.
GET /health Checks if the service is running and if the ML model has been loaded.
POST /score Main scoring endpoint. Accepts stream data and returns model predictions.

💻 Technical Details

Model Structure

The ML model file downloaded from S3 is expected to be a cloudpickle-serialized Python dictionary with the following structure:

model = {
    "preprocess": <function>,  # Function to transform request data into model input.
    "predict": <function>,     # Function to perform the actual model inference.
    "params": <dict/any>,      # Optional parameters passed to preprocess/predict.
    "stream_features": <list[str]>, # Optional list of feature names to fetch from DynamoDB.
    "kafka_features_to_ignore": <list[str]>, # Optional user/stream feature names omitted from stream-scoring Kafka logs.
}

Feature Caching (cache.py)

The ThreadSafeTLRUCache ensures that feature lookups and updates are thread-safe. The _ttu (time-to-use) policy allows features to specify their own TTL via a cache_ttl_in_seconds key in the stored value.

DynamoDB Feature Fetching (dynamo.py)

The set_stream_features function handles:

  • Checking the in-memory cache for required stream_features.

  • Batch-fetching any missing features from DynamoDB.

  • Parsing the low-level DynamoDB items into Python types.

  • Populating the cache with the fetched data, respecting the feature's TTL.

  • Injecting the fetched feature values back into the streams list in the request payload.

Metadata

Release files for haystack-ml-stack 0.4.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haystack-ml-stack 0.4.15
File Size Uploaded Provenance Actions
haystack_ml_stack-0.4.15.tar.gz 38.5 kB Without provenance Details for haystack_ml_stack-0.4.15.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for haystack-ml-stack 0.4.15
File Interpreter ABI Platform Actions
haystack_ml_stack-0.4.15-py3-none-any.whl Python 3 none any Details for haystack_ml_stack-0.4.15-py3-none-any.whl

Total release size: 69.8 kB

Release files / haystack_ml_stack-0.4.15.tar.gz

Download URL haystack_ml_stack-0.4.15.tar.gz
Size 38.5 kB
Tags Source
SHA-256 checksum
How to use checksums
5deb66e224d338d38edb82fd2beecbe2b49b4604cb6b4bc02485bed9e6c8e036
BLAKE2b-256 checksum
How to use checksums
a9b3836408856f9a8f0e13327f04ea4c1cefdcb7e26b85e5a2c69833bf13b39e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.15

Release files / haystack_ml_stack-0.4.15-py3-none-any.whl

Download URL haystack_ml_stack-0.4.15-py3-none-any.whl
Size 31.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b3bbd642ec806a547bb19732380cee9c16b248a6dd35c34cac9ed37c362e0c51
BLAKE2b-256 checksum
How to use checksums
53f1e8d2ab16034979a0b3f6dde376108bf75fc924b5cac00f895cf783618e21
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.15

Security for haystack-ml-stack 0.4.15

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.15 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page