Haystack ML Stack

Currently this project contains a FastAPI-based service designed for low-latency scoring of streams data coming from http requests

🚀 Features

FastAPI Service: Lightweight and fast web service for ML inference.
- Asynchronous I/O: Utilizes aiobotocore for non-blocking S3 and DynamoDB operations.
- Model Loading: Downloads and loads the ML model (using cloudpickle) from a configurable S3 path on startup.
- Feature Caching: Implements a thread-safe Time-To-Live (TTL) / Least-Recently-Used (LRU) cache (cachetools.TLRUCache) for DynamoDB features, reducing latency and database load.
- DynamoDB Integration: Fetches stream-specific features from DynamoDB to enrich the data before scoring.
- Health Check: Provides a /health endpoint to monitor service status and model loading.

📦 Installation

This project requires Python 3.11 or later.

Install package: The dependencies associated are listed in pyproject.toml.
```
pip install haystack-ml-stack
```

⚙️ Configuration

The service is configured using environment variables, managed by pydantic-settings. You can use a .env file for local development.

Variable Name	Alias	Default	Description
`S3_MODEL_PATH`	`S3_MODEL_PATH`	`None`	Required. The `s3://bucket/key` URL for the cloudpickled ML model file.
`FEATURES_TABLE`	`FEATURES_TABLE`	`"features"`	Name of the DynamoDB table storing stream features.
`LOGS_FRACTION`	`LOGS_FRACTION`	`0.01`	Fraction of requests to log detailed stream data for sampling/debugging (0.0 to 1.0).
`CACHE_MAXSIZE`	(none)	`50000`	Maximum size of the in-memory feature cache.

Example env vars

S3_MODEL_PATH="s3://my-ml-models/stream-scorer/latest.pkl"
FEATURES_TABLE="features"
LOGS_FRACTION=0.05

🌐 Endpoints

Method	Path	Description
GET	`/`	Root endpoint, returns a simple running message.
GET	`/health`	Checks if the service is running and if the ML model has been loaded.
POST	`/score`	Main scoring endpoint. Accepts stream data and returns model predictions.

💻 Technical Details

Model Structure

The ML model file downloaded from S3 is expected to be a cloudpickle-serialized Python dictionary with the following structure:

model = {
    "preprocess": <function>,  # Function to transform request data into model input.
    "predict": <function>,     # Function to perform the actual model inference.
    "params": <dict/any>,      # Optional parameters passed to preprocess/predict.
    "stream_features": <list[str]>, # Optional list of feature names to fetch from DynamoDB.
}

Feature Caching (cache.py)

The ThreadSafeTLRUCache ensures that feature lookups and updates are thread-safe. The _ttu (time-to-use) policy allows features to specify their own TTL via a cache_ttl_in_seconds key in the stored value.

DynamoDB Feature Fetching (dynamo.py)

The set_stream_features function handles:

Checking the in-memory cache for required stream_features.
Batch-fetching any missing features from DynamoDB.
Parsing the low-level DynamoDB items into Python types.
Populating the cache with the fetched data, respecting the feature's TTL.
Injecting the fetched feature values back into the streams list in the request payload.

Metadata

Release files for haystack-ml-stack 0.4.11

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haystack-ml-stack 0.4.11
File	Size	Uploaded
haystack_ml_stack-0.4.11.tar.gz	35.0 kB	Aug 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for haystack-ml-stack 0.4.11
File	Interpreter	ABI	Platform	Reset
haystack_ml_stack-0.4.11-py3-none-any.whl 35.7 kB Aug 10, 2026	Python 3	none	any	Details

Total release size: 70.7 kB

Release files / haystack_ml_stack-0.4.11.tar.gz

Download URL	haystack_ml_stack-0.4.11.tar.gz
Size	35.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`984d581fa5a0fae08ddac959bb1339b2a8a16a09e4226e847552f19950cfc210`
BLAKE2b-256 checksum How to use checksums	`2544c339c159088c7719834b63773db2e1557d4b647b6ec237e4d2ebd342ee06`
Upload date	Aug 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Release files / haystack_ml_stack-0.4.11-py3-none-any.whl

Download URL	haystack_ml_stack-0.4.11-py3-none-any.whl
Size	35.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a6435ccc0611f3cb964932f71e7fd862d8b5a7c833c3ffb51ce2119c244322fa`
BLAKE2b-256 checksum How to use checksums	`b42adbe5d05a678bcb8c6e8e81f8be20338d60973916e0c5aea2b1f634bc9aa0`
Upload date	Aug 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`