intersystems-iris-haystack

Table of Contents

• Overview
• Installation
• Quick Start
• Documentation
• License

---

Overview

An integration of InterSystems IRIS database with Haystack 2.x by deepset. In IRIS, the native VECTOR(DOUBLE, N) type is used for storing document embeddings, and the VECTOR_COSINE function enables high-performance dense retrievals using SIMD operations.

The library allows using InterSystems IRIS as a DocumentStore, implementing the required Protocol methods. You can start working with the implementation by importing it from the package:

from intersystems_iris_haystack.document_stores import IRISDocumentStore

In addition to the IRISDocumentStore, the library includes the following Haystack components which can be used in a pipeline:

• IRISEmbeddingRetriever - A component used to query the vector store and find semantically related Documents. It uses VECTOR_COSINE natively in the database.

• IRISBm25Retriever - A keyword-based retriever that implements Okapi BM25 over the stored documents.

The intersystems-iris-haystack library uses the official intersystems-iris Python Driver to interact with the database and hides all SQL complexities under the hood.

                                   +-----------------------------+
                                   |   InterSystems IRIS DB      |
                                   +-----------------------------+
                                   |                             |
                                   |      +----------------+     |
                                   |      |  document_table|     |
                write_documents    |      +----------------+     |
          +------------------------+----->|  id (VARCHAR)  |     |
          |                        |      |  content (CLOB)|     |
+---------+----------+             |      |  meta (JSON)   |     |
|                    |             |      |  embedding     |     |
| IRISDocumentStore  |             |      +--------+-------+     |
|                    |             |               |             |
+---------+----------+             |               |             |
          |                        |               |             |
          |                        |      +--------+--------+    |
          |                        |      | VECTOR_COSINE   |    |
          +----------------------->|      | SIMD execution  |    |
               query_embeddings    |      +-----------------+    |
                                   |                             |
                                   +-----------------------------+

In the above diagram:

• Documents are stored as rows in a dedicated relational table.
• Meta properties are stored as natively queryable JSON.
• embedding is stored as a VECTOR column type.
• Retrievals are executed by the database engine directly, eliminating the need for an external vector database.

Installation

Install the integration via pip:

pip install intersystems-iris-haystack

The package supports Haystack 2.x (haystack-ai>=2.27,<3.0). For the examples below, install the optional Sentence Transformers backend as well:

pip install "intersystems-iris-haystack" "sentence-transformers"

Requires: Python 3.10+ (Recommended/Tested on 3.12) and a running InterSystems IRIS instance.

Running InterSystems IRIS

Start IRIS locally with Docker:

docker compose -f examples/docker-compose.yaml up -d --wait --wait-timeout 180

Start an interactive terminal with the following:

docker exec -it iris-haystack iris session IRIS

Or login to the Mangement Portal at http://localhost:52773/csp/sys/%25CSP.Portal.Home.zen

The development Compose configuration creates the username demo with password demo. Use these credentials only for local development.

---

Quick start

Create a .env file using .env.example template and import the default config credentials for IntersystemsIris.

IRIS_CONNECTION_STRING="localhost:1972/USER"
IRIS_USERNAME="demo"
IRIS_PASSWORD="demo"

Example (RAG)

from haystack import Document, Pipeline
from haystack.components.embedders import (
    SentenceTransformersDocumentEmbedder,
    SentenceTransformersTextEmbedder,
)
from haystack.components.writers import DocumentWriter
from haystack.document_stores.types import DuplicatePolicy

from intersystems_iris_haystack.document_stores import IRISDocumentStore
from intersystems_iris_haystack.components.retrievers import (
    IRISEmbeddingRetriever,
    IRISBm25Retriever,
)

MODEL = "sentence-transformers/all-MiniLM-L6-v2"
store = IRISDocumentStore(embedding_dim=384)

# Indexing
indexing = Pipeline()
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder(model=MODEL))
indexing.add_component("writer", DocumentWriter(store, policy=DuplicatePolicy.OVERWRITE))
indexing.connect("embedder.documents", "writer.documents")
indexing.run(
    {
        "embedder": {
            "documents": [
                Document(content="IRIS is a multimodel database.", meta={"category": "db"}),
                Document(content="Haystack builds LLM pipelines.", meta={"category": "ai"}),
            ]
        }
    }
)

# Semantic search
query_pipeline = Pipeline()
query_pipeline.add_component("embedder", SentenceTransformersTextEmbedder(model=MODEL))
query_pipeline.add_component("retriever", IRISEmbeddingRetriever(store, top_k=3))
query_pipeline.connect("embedder.embedding", "retriever.query_embedding")

result = query_pipeline.run({"embedder": {"text": "what is vector search?"}})

# BM25 keyword search
bm25 = IRISBm25Retriever(store, top_k=3)
result = bm25.run(query="multimodel database")

---

Documentation

The full documentation is built with MkDocs Material and covers installation, all components, API reference, and a contributor guide.

Serve locally

With hatch (recommended)

# Install hatch if you don't have it
pip install hatch

# Serve docs with live reload at http://127.0.0.1:8000
hatch run docs:serve

With pip

pip install mkdocs-material mkdocstrings[python] \
mkdocs-git-revision-date-localized-plugin \
mkdocs-minify-plugin pymdown-extensions mike

mkdocs serve

---

Development

Setup

git clone https://github.com/s-c-ai/iris-haystack.git
cd iris-haystack

# Start IRIS and example
cp .env.example .env
docker compose -f examples/docker-compose.yaml up -d --wait --wait-timeout 180
hatch run example:run

# Run all tests
hatch run test:all

Test commands

Command	Description
hatch run test:unit	Unit tests — no IRIS required
hatch run test:integration	Integration tests — IRIS must be running
hatch run test:all	All tests
hatch run test:cov	All tests with coverage report

Code quality

hatch run fmt          # format and fix lint issues
hatch run fmt-check    # check only (used in CI)
hatch run type-check   # mypy

---

License

Apache 2.0 — see LICENSE.

---

References

• Haystack 2.x — Custom DocumentStore
• Haystack — Metadata Filtering
• InterSystems IRIS — Vector Search
• intersystems-irispython — DB-API driver
• InterSystems Developer Community
• Haystack Integrations